Identifying products from on-shelf sensor data and visual data

ABSTRACT

A non-transitory computer-readable medium includes instructions that when executed by a processor cause the processor to perform a method for identifying products from on-shelf sensors and image data. The method may include receiving data captured using a plurality of sensors positioned between at least part of a retail shelf and one or more products placed on the at least part of the retail shelf. The method may also include receiving an image of the at least part of the retail shelf and at least one of the one or more products. The method may also include analyzing the captured data and the image to determine a product type of the one or more products.

CROSS REFERENCES TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.17/357,692, filed on Jun. 24, 2021, which is a continuation ofInternational Application No. PCT/US2021/034885, filed May 28, 2021,which claims the benefit of priority of U.S. Provisional Application No.63/033,055, filed on Jun. 1, 2020; U.S. Provisional Application No.63/084,086, filed on Sep. 28, 2020; U.S. Provisional Application No.63/091,009, filed on Oct. 13, 2020; and U.S. Provisional Application No.63/113,490, filed on Nov. 13, 2020. The foregoing applications areincorporated herein by reference in their entirety.

BACKGROUND I. Technical Field

The present disclosure relates generally to systems, methods, anddevices for identifying products in retail stores, and more specificallyto systems, methods, and devices for capturing, collecting, andautomatically analyzing images of products displayed in retail storesfor purposes of providing one or more functions associated with theidentified products.

II. Background Information

Shopping in stores is a prevalent part of modern daily life. Storeowners (also known as “retailers”) stock a wide variety of products onstore shelves and add associated labels and promotions to the storeshelves. Typically, retailers have a set of processes and instructionsfor organizing products on the store shelves. The source of some ofthese instructions may include contractual obligations and otherpreferences related to the retailer methodology for placement ofproducts on the store shelves. Nowadays, many retailers and supplierssend people to stores to personally monitor compliance with the desiredproduct placement. Such a monitoring technique, however, may beinefficient and may result in nonuniform compliance among retailersrelative to various product-related guidelines. This technique may alsoresult in significant gaps in compliance, as it does not allow forcontinuous monitoring of dynamically changing product displays. Toincrease productivity, among other potential benefits, there is atechnological need to provide a dynamic solution that will automaticallymonitor retail spaces. Such a solution, for example and among otherfeatures, may automatically determine whether a disparity exists betweena desired product placement and an actual product placement.

The disclosed devices and methods are directed to providing new ways formonitoring retail establishments using image processing and supportingsensors.

SUMMARY

Embodiments consistent with the present disclosure provide systems,methods, and devices for capturing, collecting, and analyzing images ofproducts displayed in retail stores. For example, consistent with thedisclosed embodiments, an example system may receive an image depictinga store shelf having products displayed thereon, identify the productson the store shelf, and trigger an alert when disparity exists betweenthe desired product placement and the actual product placement.

Another aspect of the present disclosure is directed to a computerprogram product for identifying products and monitoring planogramcompliance using analysis of image data embodied in a non-transitorycomputer-readable medium and executable by at least one processor, thecomputer program product including instructions for causing the at leastone processor to execute the method described above.

In an embodiment, a non-transitory computer-readable medium includinginstructions that when executed by a processor cause the processor toperform a method for planning deployment of image sensors. The methodmay include determining a location of a store shelf within a retailstore and obtaining a first coverage parameter corresponding to a firstproduct type and a second coverage parameter corresponding to a secondproduct type. The method may also include accessing a database todetermine a first height of products of the first product type and asecond height of products of the second product type. The method mayfurther include determining a position for placing a camera configuredto capture images of at least a portion of the store shelf by analyzingthe location of the store shelf, the first coverage parameter, thesecond coverage parameter, the first height, and the second height. Themethod may also include providing, to a user interface of a user device,information relating to the determined position of the camera.

In an embodiment, a system for planning deployment of image sensors mayinclude at least one processor programmed to determine a location of astore shelf within a retail store. The at least one processor may alsoprogrammed to obtain a first coverage parameter corresponding to a firstproduct type and a second coverage parameter corresponding to a secondproduct type. The at least one processor may further programmed toaccess a database to determine a first height of products of the firstproduct type and a second height of products of the second product type.The at least one processor may also programmed to determine a positionfor placing a camera configured to capture images of at least a portionof the store shelf by analyzing the location of the store shelf, thefirst coverage parameter, the second coverage parameter, the firstheight, and the second height. The at least one processor may furtherprogrammed to provide, to a user interface of a user device, informationrelating to the determined position of the camera.

In an embodiment, a method for planning deployment of image sensors mayinclude determining a location of a store shelf within a retail store.The method may also include obtaining a first coverage parametercorresponding to a first product type and a second coverage parametercorresponding to a second product type. The method may further includeaccessing a database to determine a first height of products of thefirst product type and a second height of products of the second producttype. The method may also include determining a position for placing acamera configured to capture images of at least a portion of the storeshelf by analyzing the location of the store shelf, the first coverageparameter, the second coverage parameter, the first height, and thesecond height. The method may further include providing, to a userinterface of a user device, information relating to the determinedposition of the camera.

In an embodiment, a non-transitory computer-readable medium includinginstructions that when executed by a processor cause the processor toperform a method for routing a cleaning robot. The method may includereceiving an indication of at least one store shelf of a retail store.The method may also include causing a first adjustment to a cleaningrobot route of the cleaning robot within the retail store based on atleast one location within the retail store corresponding to the at leastone store shelf. The first adjustment may be configured to enable animage sensor associated with the cleaning robot to capture one or moreimages of at least one portion of the at least one store shelf. Themethod may further include receiving a first image acquired by the imagesensor associated with the cleaning robot. The first image may include arepresentation of the at least one portion of the at least one storeshelf. The method may also include analyzing the first image todetermine a need for a second image of the at least one portion of theat least one store shelf. The method may further include causing asecond adjustment to the cleaning robot route within the retail store,in response to the determined need. The second adjustment may beconfigured to enable the image sensor to capture the second image of theat least one portion of the at least one store shelf.

In an embodiment, a system for planning deployment of image sensors mayinclude at least one processor programmed to receive an indication of atleast one store shelf of a retail. The at least one processor may alsoprogrammed to cause a first adjustment to a cleaning robot route of thecleaning robot within the retail store based on at least one locationwithin the retail store corresponding to the at least one store shelf.The first adjustment may be configured to enable an image sensorassociated with the cleaning robot to capture one or more images of atleast one portion of the at least one store shelf. The at least oneprocessor may further programmed to receive a first image acquired bythe image sensor associated with the cleaning robot. The first image mayinclude a representation of the at least one portion of the at least onestore shelf. The at least one processor may also programmed to analyzethe first image to determine a need for a second image of the at leastone portion of the at least one store shelf. The at least one processormay further programmed to cause a second adjustment to the cleaningrobot route within the retail store, in response to the determined need.The second adjustment may be configured to enable the image sensor tocapture the second image of the at least one portion of the at least onestore shelf.

In an embodiment, a method for routing a cleaning robot may includereceiving an indication of at least one store shelf of a retail store.The method may also include causing a first adjustment to a cleaningrobot route of the cleaning robot within the retail store based on atleast one location within the retail store corresponding to the at leastone store shelf. The first adjustment may be configured to enable animage sensor associated with the cleaning robot to capture one or moreimages of at least one portion of the at least one store shelf. Themethod may further include receiving a first image acquired by the imagesensor associated with the cleaning robot. The first image may include arepresentation of the at least one portion of the at least one storeshelf. The method may also include analyzing the first image todetermine a need for a second image of the at least one portion of theat least one store shelf. The method may further include causing asecond adjustment to the cleaning robot route within the retail store,in response to the determined need. The second adjustment may beconfigured to enable the image sensor to capture the second image of theat least one portion of the at least one store shelf.

In an embodiment, a non-transitory computer-readable medium includesinstructions that when executed by a processor cause the processor toperform a method for reacting to changes of items hanging on peg-hooksconnected to pegboards. The method may include receiving a firstindication indicative of a change of items hanging on a first peg-hookconnected to a pegboard; receiving a second indication indicative of achange of items hanging on a second peg-hook connected to the pegboard;in response to the first indication, causing an action related to afirst product type; and in response to the second indication, causing anaction related to a second product type, the second product typediffering from the first product type.

In an embodiment, a method for reacting to changes of items hanging onpeg-hooks connected to pegboards may include receiving a firstindication indicative of a change of items hanging on a first peg-hookconnected to a pegboard; receiving a second indication indicative of achange of items hanging on a second peg-hook connected to the pegboard;in response to the first indication, causing an action related to afirst product type; and in response to the second indication, causing anaction related to a second product type, the second product type differsfrom the first product type.

In an embodiment, a system for reacting to changes of items hanging onpeg-hooks connected to pegboards may include at least one processorprogrammed to receive a first indication indicative of a change of itemshanging on a first peg-hook connected to a pegboard; receive a secondindication indicative of a change of items hanging on a second peg-hookconnected to the pegboard; in response to the first indication, cause anaction related to a first product type; and in response to the secondindication, cause an action related to a second product type, the secondproduct type differs from the first product type.

In an embodiment, a non-transitory computer-readable medium includesinstructions that when executed by a processor cause the processor toperform a method for identifying products from on-shelf sensor data andimage data. The method may include receiving data captured using aplurality of sensors positioned between at least part of a retail shelfand one or more products placed on the at least part of the retailshelf; receiving an image of the at least part of the retail shelf andat least one of the one or more products; and analyzing the captureddata and the image to determine a product type of the one or moreproducts.

In an embodiment, a method for identifying products from on-shelf sensordata and image data may include receiving data captured using aplurality of sensors positioned between at least part of a retail shelfand one or more products placed on the at least part of the retailshelf; receiving an image of the at least part of the retail shelf andat least one of the one or more products; and analyzing the captureddata and the image to determine a product type of the one or moreproducts.

In an embodiment, a system for identifying products from on-shelf sensordata and image data may include at least one processor programmed toreceive data captured using a plurality of sensors positioned between atleast part of a retail shelf and one or more products placed on the atleast part of the retail shelf; receive an image of the at least part ofthe retail shelf and at least one of the one or more products; andanalyze the captured data and the image to determine a product type ofthe one or more products.

In an embodiment, a method for providing visual navigation assistance inretail stores may comprise receiving a first indoor location of a userwithin a retail store; receiving a target destination within the retailstore; providing first navigation data to the user through a firstvisual interface; after providing the first navigation data, receiving asecond indoor location of the user within the retail store; determiningthat the second indoor location is within a selected area around thetarget destination, the selected area not including the first indoorlocation; and in response to the determination that the second indoorlocation is within the selected area around the target destination,providing second navigation data to the user through a second visualinterface, the second visual interface differing from the first visualinterface.

In an embodiment, a computer program product may identify products andmonitoring planogram compliance using analysis of image data embodied ina non-transitory computer-readable medium and executable by at least oneprocessor. The computer program product may include instructions forcausing the at least one processor to execute the method describedabove.

In an embodiment, a method for making gradual adjustments to planogramsis disclosed. The method may comprise receiving a first image of atleast part of a shelf; analyzing the first image to determine a firstplacement of products on the at least part of the shelf; based on thedetermined first placement of products, determining a planned firstadjustment to the determined first placement of products on the at leastpart of the shelf; and based on the planned first adjustment to thedetermined first placement of products, providing first information, thefirst information being configured to cause the planned first adjustmentto the determined first placement of products. The method may furthercomprise receiving a second image of the at least part of the shelfcaptured after the first information was provided; analyzing the secondimage to determine a second placement of products on the at least partof the shelf; based on the determined second placement of products,determining a planned second adjustment to the determined secondplacement of products on the at least part of the shelf; and based onthe planned second adjustment to the determined second placement ofproducts, providing second information, the second information beingconfigured to cause the planned second adjustment to the determinedsecond placement of products.

In an embodiment, a system for making gradual adjustments to planogramscomprise at least one processor. The at least one processor may beprogrammed to receive a first image of at least part of a shelf; analyzethe first image to determine a first placement of products on the atleast part of the shelf; based on the determined first placement ofproducts, determine a planned first adjustment to a determined firstplacement of products on the at least part of the shelf; and based onthe planned first adjustment to the determined first placement ofproducts, provide first information, the first information beingconfigured to cause the planned first adjustment to the determined firstplacement of products. The processor may further be programmed toreceive a second image of the at least part of the shelf captured afterthe first information was provided; analyze the second image todetermine a second placement of products on the at least part of theshelf; based on the determined second placement of products, determine aplanned second adjustment to the determined second placement of productson the at least part of the shelf; and based on the planned secondadjustment to the determined second placement of products, providesecond information, the second information being configured to cause theplanned second adjustment to the determined second placement ofproducts.

In an embodiment, a method for testing of planograms is disclosed. Themethod may comprise receiving a first image of at least part of a shelf;analyzing the first image to determine a first placement of products onthe at least part of the shelf; based on the first placement ofproducts, determining a planned adjustment to the first placement ofproducts on the at least part of the shelf; and generating firstinstructions to implement the planned first adjustment. The method mayfurther comprise receiving a second image of the at least part of theshelf captured after the first instructions were generated; analyzingthe second image to determine a second placement of products on the atleast part of the shelf, the second placement of products resulting fromthe planned first adjustment; receiving an indication of an impact ofthe second placement of products; determining a planned secondadjustment to the second placement of products on the at least part ofthe shelf, the planned second adjustment being determined based on theimpact; and generating second instructions to implement the plannedsecond adjustment.

In an embodiment, a system for testing of planograms may comprise atleast one processor. The at least one processor may be programmed toreceive a first image of at least part of a shelf; analyze the firstimage to determine a first placement of products on the at least part ofthe shelf; based on the first placement of products, determine a plannedfirst adjustment to the first placement of products on the at least partof the shelf; and generate first instructions to implement the plannedfirst adjustment to the first placement of products. The at least oneprocessor may further be programmed to receive a second image of the atleast part of the shelf captured after the first instructions weregenerated; analyze the second image to determine a second placement ofproducts on the at least part of the shelf, the second placement ofproducts resulting from the planned first adjustment; receive anindication of an impact of the second placement of products; determine aplanned second adjustment to the second placement of products on the atleast part of the shelf, the planned second adjustment being determinedbased on the impact; and generate second instructions to implement theplanned second adjustment.

In an embodiment, a method for assessing quality of tasks performed bypersons in retail stores may comprise receiving an indication that aperson completed a task corresponding to at least one shelf in a retailstore; receiving at least one image of the at least one shelf capturedusing an image sensor after the completion of the task; analyzing the atleast one image to determine at least one property associated withperforming the task; and using the at least one property to determine areward for performing the task.

In an embodiment, a system for assessing quality of tasks performed bypersons in retail stores include at least one processor; and anon-transitory computer-readable medium containing instructions that,when executed by the at least one processor, cause the at least oneprocessor to: receive an indication that a person completed a taskcorresponding to at least one shelf in a retail store; receive at leastone image of the at least one shelf captured using an image sensor afterthe completion of the task; analyze the at least one image to determineat least one property of associated with performing the task; and usethe at least one property to determine a reward for performing the task.

In an embodiment, a computer program product may assess quality of tasksperformed by persons in retail stores embodied in a non-transitorycomputer-readable medium and executable by at least one processor. Thecomputer program product may include instructions for causing the atleast one processor to execute the method described above.

In some embodiments, an indication that a person completed a taskcorresponding to at least one shelf in a retail store may be received,and at least one image of the at least one shelf may be received, the atleast one image being captured using an image sensor after thecompletion of the task. The at least one image may be analyzed todetermine at least one property associated with performing the task, andthe at least one property may be used to determine a reward forperforming the task. For example, the indication that the personcompleted the task may be based on an input from the person, may bebased on an analysis of the at least one image, and so forth. In oneexample, the at least one property associated with performing the taskmay include a quality indication of at least one aspect of performingthe task, and the determination of the reward may be based on thequality indication. In one example, the at least one image may be atleast one image captured by at least one of the person or an imagesensor mounted to a shelf. In one example, the at least one image may beanalyzed to determine a property of the person, and the determination ofthe reward may be based on the property of the person. In one example,the task may include at least one of a restocking of a productassociated with the at least one shelf, a correction of product facingsat the at least one shelf, removing a product from the at least oneshelf, changing a price of at least one product, and changing anorientation of at least one product on the at least one shelf. In oneexample, the at least one image may be analyzed to determine at leastone additional available task corresponding to the at least one shelf inthe retail store, and an indication of the additional available task maybe provided to the person. In one example, the at least one image may beanalyzed to determine that the person performed a positive actioncorresponding to the at least one shelf in the retail store (thepositive action may not be included in the task), and the determinationof the reward may be based on the positive action. In one example, animpact of the performed task may be determined, and the determination ofthe reward may be based on the impact. In one example, the task mayinclude at least one of positioning or removing a promotional sign atthe at least one shelf. In one example, the at least one image may beanalyzed to determine at least one aspect lacking in the performance ofthe task, an indication of the at least one aspect may be provided tothe person, at least one additional image of the at least one shelf maybe received (the at least one image may be at least one image beingcaptured using the image sensor after the indication of the at least oneaspect is provided), and the at least one additional image may beanalyzed to determine the reward for performing the task. In oneexample, the reward may include a monetary reward, and the determinationof the reward may include a determination of an amount associated withthe monetary reward. In one example, input from at least one pressuresensor positioned on the at least one shelf may be received, and thedetermination of the at least one property may be based on an analysisof the received input. In one example, input from at least one touchsensor positioned on the at least one shelf may be received, and thedetermination of the at least one property may be based on an analysisof the received input. In one example, input from at least one weightsensor positioned on the at least one shelf may be received, and thedetermination of the at least one property may be based on an analysisof the received input. In one example, input from at least one lightsensor positioned on the at least one shelf may be received, and thedetermination of the at least one property may be based on an analysisthe received input. In one example, analyzing the at least one image todetermine the at least one property may comprise identifying at leastone visual indicator in the at least one image (the visual indicator maycomprise at least one of: a color, a brightness, a character, and ashape), comparing the at least one visual indicator to a referencevisual indicator, and determining a degree of similarity between theidentified at least one visual indicator and the reference visualindicator.

In an embodiment, a non-transitory computer-readable medium may includeinstructions that when executed by a processor cause the processor toperform a method for selecting available assignments for users based onmobile computing devices of the users, the method comprising: receivingan indication of at least one parameter of a mobile computing deviceassociated with a user; accessing a plurality of available assignments,each assignment of the plurality of available assignments involving useof at least one mobile computing device; based on the at least oneparameter of the mobile computing device associated with the user,selecting a subset of the plurality of available assignments; andoffering the selected subset of the plurality of available assignmentsto the user.

In an embodiment, a method for selecting available assignments for usersbased on mobile computing devices of the users may include receiving anindication of at least one parameter of a mobile computing deviceassociated with a user; accessing a plurality of available assignments,each assignment of the plurality of available assignments involving useof at least one mobile computing device; based on the at least oneparameter of the mobile computing device associated with the user,selecting a subset of the plurality of available assignments; andoffering the selected subset of the plurality of available assignmentsto the user.

In an embodiment, a system for selecting available assignments for usersbased on mobile computing devices of the users may include at least oneprocessor programmed to: receive an indication of at least one parameterof a mobile computing device associated with a user; access a pluralityof available assignments, each assignment of the plurality of availableassignments involving use of at least one mobile computing device; basedon the at least one parameter of the mobile computing device associatedwith the user, select a subset of the plurality of availableassignments; and offer the selected subset of the plurality of availableassignments to the user.

In an embodiment, a non-transitory computer-readable medium may includeinstructions that when executed by a processor cause the processor toperform a method for selecting available assignments, the methodcomprising: receiving an indication of an external assignment, whereinthe external assignment is associated with one or more locationsexternal to a plurality of retail stores; based on the indication of theexternal assignment, selecting a retail store among the plurality ofretail stores; based on the selected retail store, selecting at leastone available assignment in the selected retail store; and offering theselected at least one available assignment in the selected retail storeto a user.

In an embodiment, a method for selecting available assignments mayinclude receiving an indication of an external assignment, wherein theexternal assignment is associated with one or more locations external toa plurality of retail stores; based on the indication of the externalassignment, selecting a retail store among the plurality of retailstores; based on the selected retail store, selecting at least oneavailable assignment in the selected retail store; and offering theselected at least one available assignment in the selected retail storeto a user.

In an embodiment, a system for selecting available assignments in retailstores for users based on external assignments may include: at least oneprocessor programmed to: receive an indication of an externalassignment, wherein the external assignment is associated with one ormore locations external to a plurality of retail stores; based on theindication of the external assignment, select a retail store among theplurality of retail stores; based on the selected retail store, selectat least one available assignment in the selected retail store; andoffering the selected at least one available assignment in the selectedretail store to a user.

Consistent with other disclosed embodiments, non-transitorycomputer-readable storage media may store program instructions, whichare executed by at least one processing device and perform any of themethods described herein.

The foregoing general description and the following detailed descriptionare exemplary and explanatory only and are not restrictive of theclaims.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this disclosure, illustrate various disclosed embodiments. Inthe drawings:

FIG. 1 is an illustration of an exemplary system for analyzinginformation collected from a retail store.

FIG. 2 is a block diagram that illustrates some of the components of animage processing system, consistent with the present disclosure.

FIG. 3 is a block diagram that illustrates an exemplary embodiment of acapturing device, consistent with the present disclosure.

FIG. 4A is a schematic illustration of an example configuration forcapturing image data in a retail store, consistent with the presentdisclosure.

FIG. 4B is a schematic illustration of another example configuration forcapturing image data in a retail store, consistent with the presentdisclosure.

FIG. 4C is a schematic illustration of another example configuration forcapturing image data in a retail store, consistent with the presentdisclosure.

FIG. 5A is an illustration of an example system for acquiring images ofproducts in a retail store, consistent with the present disclosure.

FIG. 5B is an illustration of a shelf-mounted camera unit included in afirst housing of the example system of FIG. 5A, consistent with thepresent disclosure.

FIG. 5C is an exploded view illustration of a processing unit includedin a second housing of the example system of FIG. 5A, consistent withthe present disclosure.

FIG. 6A is a top view representation of an aisle in a retail store withmultiple image acquisition systems deployed thereon for acquiring imagesof products, consistent with the present disclosure.

FIG. 6B is a perspective view representation of part of a retailshelving unit with multiple image acquisition systems deployed thereonfor acquiring images of products, consistent with the presentdisclosure.

FIG. 6C provides a diagrammatic representation of how the exemplarydisclosed image acquisition systems may be positioned relative to retailshelving to acquire product images, consistent with the presentdisclosure.

FIG. 7A provides a flowchart of an exemplary method for acquiring imagesof products in retail store, consistent with the present disclosure.

FIG. 7B provides a flowchart of a method for acquiring images ofproducts in retail store, consistent with the present disclosure.

FIG. 8A is a schematic illustration of an example configuration fordetecting products and empty spaces on a store shelf, consistent withthe present disclosure.

FIG. 8B is a schematic illustration of another example configuration fordetecting products and empty spaces on a store shelf, consistent withthe present disclosure.

FIG. 9 is a schematic illustration of example configurations fordetection elements on store shelves, consistent with the presentdisclosure.

FIG. 10A illustrates an exemplary method for monitoring planogramcompliance on a store shelf, consistent with the present disclosure.

FIG. 10B is illustrates an exemplary method for triggering imageacquisition based on product events on a store shelf, consistent withthe present disclosure.

FIG. 11A is a schematic illustration of an example output for a marketresearch entity associated with the retail store, consistent with thepresent disclosure.

FIG. 11B is a schematic illustration of an example output for a supplierof the retail store, consistent with the present disclosure.

FIG. 11C is a schematic illustration of an example output for a managerof the retail store, consistent with the present disclosure.

FIG. 11D is a schematic illustration of two examples outputs for a storeassociate of the retail store, consistent with the present disclosure.

FIG. 11E is a schematic illustration of an example output for an onlinecustomer of the retail store, consistent with the present disclosure.

FIGS. 12A, 12B, and 12C are schematic illustration of exemplarypositions of cameras relative to store shelves.

FIG. 13 is a block diagram of an exemplary system for planningdeployment of image sensors.

FIG. 14 provides a flowchart of an exemplary method for planningdeployment of an image sensor.

FIG. 15 is a schematic illustration of an example cleaning robot,consistent with the present disclosure.

FIG. 16 is a schematic illustration of an example cleaning robot,consistent with the present disclosure.

FIG. 17A is a block diagram of an example system for navigating acleaning robot, consistent with the present disclosure.

FIG. 17B provides an exemplary process for navigating a cleaning robot,consistent with the present disclosure.

FIG. 18A is an illustration of an example pegboard having a plurality ofpeg-hooks for hanging items, consistent with the present disclosure.

FIG. 18B is an illustration of an example sensor for sensing itemshanging on a peg-hook, consistent with the present disclosure.

FIG. 18C is an illustration of another example sensor for sensing itemshanging on a peg-hook, consistent with the present disclosure.

FIG. 18D is an illustration of another example sensor for sensing itemshanging on a peg-hook, consistent with the present disclosure.

FIG. 18E is an illustration of another example sensor for sensing itemshanging on a peg-hook, consistent with the present disclosure.

FIG. 18F is an illustration of an example for transmitting signalsgenerated by peg-hook sensors to a computer for processing, consistentwith the present disclosure.

FIG. 18G is an illustration of an example distribution map of itemshanging on a pegboard, consistent with the present disclosure.

FIG. 19 is an illustration of an example system for monitoring changesof items hanging on peg-hooks connected to a pegboard, consistent withthe present disclosure.

FIG. 20 provides a flowchart of an exemplary method for detectingchanges of items hanging on peg-hooks connected to a pegboard,consistent with the present disclosure.

FIG. 21 is an illustration of a sensor fusion technique for determiningproduct type, consistent with the present disclosure.

FIG. 22 is an illustration of an example system for identifying productson a retail shelf, consistent with the present disclosure.

FIG. 23 provides a flowchart of an exemplary method for identifyingproducts on a retail shelf, consistent with the present disclosure.

FIG. 24 is a visual depiction of an exemplary navigation assistancemap-view user interface, consistent with the present disclosure.

FIG. 25 is a visual depiction of an exemplary navigation assistanceaugmented-reality-view user interface, consistent with the presentdisclosure.

FIG. 26 is a flowchart of exemplary process for providing visualnavigation assistance in retail stores, consistent with the presentdisclosure.

FIG. 27A illustrates an example image including at least part of ashelf, consistent with the present disclosure.

FIG. 27B illustrates a target planogram that may be received, consistentwith the present disclosure.

FIGS. 28A and 28B illustrate example adjustments to product placements,consistent with the present disclosure.

FIG. 28C illustrates an example image showing a failed execution of aplanned adjustment, consistent with the present disclosure.

FIG. 28D illustrates example modified adjustments that may be determinedbased on subsequent images, consistent with the present disclosure.

FIG. 29 provides a flowchart of an exemplary method for making gradualadjustments to planograms, consistent with the present disclosure.

FIG. 30A illustrates an example image including at least part of ashelf, consistent with the present disclosure.

FIG. 30B illustrates an example adjustment that may focus onrearrangement of the top portion of shelf, consistent with the presentdisclosure.

FIG. 31A illustrates an additional example image captured afterinstructions for implementing an adjustment were generated, consistentwith the present disclosure.

FIG. 31B illustrates example data that may indicate an impact of anadjusted product placement, consistent with the present disclosure.

FIG. 31C illustrates another example adjustment that may be generatedbased on a determined impact, consistent with the present disclosure.

FIG. 32 provides a flowchart of an exemplary method for testing ofplanograms, consistent with the present disclosure.

FIG. 33 is a visual depiction of an exemplary shelf view, consistentwith the present disclosure.

FIG. 34 is a flowchart of an exemplary process for providing a rewardbased on image analysis, consistent with the present disclosure.

FIG. 35 is a flowchart of exemplary process for performing imageanalysis, consistent with the present disclosure.

FIG. 36 is a schematic illustration a communications network supportingmultiple users, consistent with the present disclosure.

FIG. 37 is a schematic illustration of a user interface, consistent withthe present disclosure.

FIG. 38 provides a flowchart of an exemplary method for providingassignments to users, consistent with the present disclosure.

FIG. 39 is a schematic illustration a communications network supportingmultiple users, consistent with the present disclosure.

FIGS. 40A and 40B are illustrations of selecting retail stores based ona route, consistent with the present disclosure.

FIG. 41 provides a flowchart of an exemplary method for selectingavailable assignments, consistent with the present disclosure.

DETAILED DESCRIPTION

The following detailed description refers to the accompanying drawings.Wherever possible, the same reference numbers are used in the drawingsand the following description to refer to the same or similar parts.While several illustrative embodiments are described herein,modifications, adaptations and other implementations are possible. Forexample, substitutions, additions, or modifications may be made to thecomponents illustrated in the drawings, and the illustrative methodsdescribed herein may be modified by substituting, reordering, removing,or adding steps to the disclosed methods. Accordingly, the followingdetailed description is not limited to the disclosed embodiments andexamples. Instead, the proper scope is defined by the appended claims.

The present disclosure is directed to systems and methods for processingimages captured in a retail store. As used herein, the term “retailstore” or simply “store” refers to an establishment offering productsfor sale by direct selection by customers physically or virtuallyshopping within the establishment. The retail store may be anestablishment operated by a single retailer (e.g., supermarket) or anestablishment that includes stores operated by multiple retailers (e.g.,a shopping mall). Embodiments of the present disclosure includereceiving an image depicting a store shelf having at least one productdisplayed thereon. As used herein, the term “store shelf” or simply“shelf” refers to any suitable physical structure which may be used fordisplaying products in a retail environment. In one embodiment the storeshelf may be part of a shelving unit including a number of individualstore shelves. In another embodiment, the store shelf may include adisplay unit having a single-level or multi-level surfaces.

Consistent with the present disclosure, the system may process imagesand image data acquired by a capturing device to determine informationassociated with products displayed in the retail store. The term“capturing device” refers to any device configured to acquire image datarepresentative of products displayed in the retail store. Examples ofcapturing devices may include a digital camera, a time-of-flight camera,a stereo camera, an active stereo camera, a depth camera, a Lidarsystem, a laser scanner, CCD based devices, or any other sensor basedsystem capable of converting received light into electric signals. Theterm “image data” refers to any form of data generated based on opticalsignals in the near-infrared, infrared, visible, and ultravioletspectrums (or any other suitable radiation frequency range). Consistentwith the present disclosure, the image data may include pixel datastreams, digital images, digital video streams, data derived fromcaptured images, and data that may be used to construct a 3D image. Theimage data acquired by a capturing device may be transmitted by wired orwireless transmission to a remote server. In one embodiment, thecapturing device may include a stationary camera with communicationlayers (e.g., a dedicated camera fixed to a store shelf, a securitycamera, etc.). Such an embodiment is described in greater detail belowwith reference to FIG. 4A. In another embodiment, the capturing devicemay include a handheld device (e.g., a smartphone, a tablet, a mobilestation, a personal digital assistant, a laptop, and more) or a wearabledevice (e.g., smart glasses, a smartwatch, a clip-on camera). Such anembodiment is described in greater detail below with reference to FIG.4B. In another embodiment, the capturing device may include a roboticdevice with one or more cameras operated remotely or autonomously (e.g.,an autonomous robotic device, a drone, a robot on a track, and more).Such an embodiment is described in greater detail below with referenceto FIG. 4C.

In some embodiments, the capturing device may include one or more imagesensors. The term “image sensor” refers to a device capable of detectingand converting optical signals in the near-infrared, infrared, visible,and ultraviolet spectrums into electrical signals. The electricalsignals may be used to form image data (e.g., an image or a videostream) based on the detected signal. Examples of image sensors mayinclude semiconductor charge-coupled devices (CCD), active pixel sensorsin complementary metal-oxide-semiconductor (CMOS), or N-typemetal-oxide-semiconductors (NMOS, Live MOS). In some cases, the imagesensor may be part of a camera included in the capturing device.

Embodiments of the present disclosure further include analyzing imagesto detect and identify different products. As used herein, the term“detecting a product” may broadly refer to determining an existence ofthe product. For example, the system may determine the existence of aplurality of distinct products displayed on a store shelf. By detectingthe plurality of products, the system may acquire different detailsrelative to the plurality of products (e.g., how many products on astore shelf are associated with a same product type), but it does notnecessarily gain knowledge of the type of product. In contrast, the term“identifying a product” may refer to determining a unique identifierassociated with a specific type of product that allows inventorymanagers to uniquely refer to each product type in a product catalogue.Additionally or alternatively, the term “identifying a product” mayrefer to determining a unique identifier associated with a specificbrand of products that allows inventory managers to uniquely refer toproducts, e.g., based on a specific brand in a product catalogue.Additionally or alternatively, the term “identifying a product” mayrefer to determining a unique identifier associated with a specificcategory of products that allows inventory managers to uniquely refer toproducts, e.g., based on a specific category in a product catalogue. Insome embodiments, the identification may be made based at least in parton visual characteristics of the product (e.g., size, shape, logo, text,color, etc.). The unique identifier may include any codes that may beused to search a catalog, such as a series of digits, letters, symbols,or any combinations of digits, letters, and symbols. Consistent with thepresent disclosure, the terms “determining a type of a product” and“determining a product type” may also be used interchangeably in thisdisclosure with reference to the term “identifying a product.”

Embodiments of the present disclosure further include determining atleast one characteristic of the product for determining the type of theproduct. As used herein, the term “characteristic of the product” refersto one or more visually discernable features attributed to the product.Consistent with the present disclosure, the characteristic of theproduct may assist in classifying and identifying the product. Forexample, the characteristic of the product may be associated with theornamental design of the product, the size of the product, the shape ofthe product, the colors of the product, the brand of the product, a logoor text associated with the product (e.g., on a product label), andmore. In addition, embodiments of the present disclosure further includedetermining a confidence level associated with the determined type ofthe product. The term “confidence level” refers to any indication,numeric or otherwise, of a level (e.g., within a predetermined range)indicative of an amount of confidence the system has that the determinedtype of the product is the actual type of the product. For example, theconfidence level may have a value between 1 and 10, alternatively, theconfidence level may be expressed as a percentage.

In some cases, the system may compare the confidence level to athreshold. The term “threshold” as used herein denotes a referencevalue, a level, a point, or a range of values, for which, when theconfidence level is above it (or below it depending on a particular usecase), the system may follow a first course of action and, when theconfidence level is below it (or above it depending on a particular usecase), the system may follow a second course of action. The value of thethreshold may be predetermined for each type of product or may bedynamically selected based on different considerations. In oneembodiment, when the confidence level associated with a certain productis below a threshold, the system may obtain contextual information toincrease the confidence level. As used herein, the term “contextualinformation” (or “context”) refers to any information having a direct orindirect relationship with a product displayed on a store shelf. In someembodiments, the system may retrieve different types of contextualinformation from captured image data and/or from other data sources. Insome cases, contextual information may include recognized types ofproducts adjacent to the product under examination. In other cases,contextual information may include text appearing on the product,especially where that text may be recognized (e.g., via OCR) andassociated with a particular meaning. Other examples of types ofcontextual information may include logos appearing on the product, alocation of the product in the retail store, a brand name of theproduct, a price of the product, product information collected frommultiple retail stores, product information retrieved from a catalogassociated with a retail store, etc.

Reference is now made to FIG. 1, which shows an example of a system 100for analyzing information collected from retail stores 105 (for example,retail store 105A, retail store 105B, and retail store 105C). In oneembodiment, system 100 may represent a computer-based system that mayinclude computer system components, desktop computers, workstations,tablets, handheld computing devices, memory devices, and/or internalnetwork(s) connecting the components. System 100 may include or beconnected to various network computing resources (e.g., servers,routers, switches, network connections, storage devices, etc.) necessaryto support the services provided by system 100. In one embodiment,system 100 may enable identification of products in retail stores 105based on analysis of captured images. In another embodiment, system 100may enable a supply of information based on analysis of captured imagesto a market research entity 110 and to different suppliers 115 of theidentified products in retail stores 105 (for example, supplier 115A,supplier 115B, and supplier 115C). In another embodiment, system 100 maycommunicate with a user 120 (sometimes referred to herein as a customer,but which may include individuals associated with a retail environmentother than customers, such as store associates, data collection agent,etc.) about different products in retail stores 105. In one example,system 100 may receive images of products captured by user 120. Inanother example, system 100 may provide to user 120 informationdetermined based on automatic machine analysis of images captured by oneor more capturing devices 125 associated with retail stores 105.

System 100 may also include an image processing unit 130 to execute theanalysis of images captured by the one or more capturing devices 125.Image processing unit 130 may include a server 135 operatively connectedto a database 140. Image processing unit 130 may include one or moreservers connected by a communication network, a cloud platform, and soforth. Consistent with the present disclosure, image processing unit 130may receive raw or processed data from capturing device 125 viarespective communication links, and provide information to differentsystem components using a network 150. Specifically, image processingunit 130 may use any suitable image analysis technique including, forexample, object recognition, object detection, image segmentation,feature extraction, optical character recognition (OCR), object-basedimage analysis, shape region techniques, edge detection techniques,pixel-based detection, artificial neural networks, convolutional neuralnetworks, etc. In addition, image processing unit 130 may useclassification algorithms to distinguish between the different productsin the retail store. In some embodiments, image processing unit 130 mayutilize suitably trained machine learning algorithms and models toperform the product identification. Network 150 may facilitatecommunications and data exchange between different system componentswhen these components are coupled to network 150 to enable output ofdata derived from the images captured by the one or more capturingdevices 125. In some examples, the types of outputs that imageprocessing unit 130 may generate may include identification of products,indicators of product quantity, indicators of planogram compliance,indicators of service-improvement events (e.g., a cleaning event, arestocking event, a rearrangement event, etc.), and various reportsindicative of the performances of retail stores 105. Additional examplesof the different outputs enabled by image processing unit 130 aredescribed below with reference to FIGS. 11A-11E and throughout thedisclosure.

Consistent with the present disclosure, network 150 may be any type ofnetwork (including infrastructure) that provides communications,exchanges information, and/or facilitates the exchange of informationbetween the components of system 100. For example, network 150 mayinclude or be part of the Internet, a Local Area Network, wirelessnetwork (e.g., a Wi-Fi/302.11 network), or other suitable connections.In other embodiments, one or more components of system 100 maycommunicate directly through dedicated communication links, such as, forexample, a telephone network, an extranet, an intranet, the Internet,satellite communications, off-line communications, wirelesscommunications, transponder communications, a local area network (LAN),a wide area network (WAN), a virtual private network (VPN), and soforth.

In one example configuration, server 135 may be a cloud server thatprocesses images received directly (or indirectly) from one or morecapturing device 125 and processes the images to detect and/or identifyat least some of the plurality of products in the image based on visualcharacteristics of the plurality of products. The term “cloud server”refers to a computer platform that provides services via a network, suchas the Internet. In this example configuration, server 135 may usevirtual machines that may not correspond to individual hardware. Forexample, computational and/or storage capabilities may be implemented byallocating appropriate portions of desirable computation/storage powerfrom a scalable repository, such as a data center or a distributedcomputing environment. In one example, server 135 may implement themethods described herein using customized hard-wired logic, one or moreApplication Specific Integrated Circuits (ASICs) or Field ProgrammableGate Arrays (FPGAs), firmware, and/or program logic which, incombination with the computer system, cause server 135 to be aspecial-purpose machine.

In another example configuration, server 135 may be part of a systemassociated with a retail store that communicates with capturing device125 using a wireless local area network (WLAN) and may provide similarfunctionality as a cloud server. In this example configuration, server135 may communicate with an associated cloud server (not shown) andcloud database (not shown). The communications between the store serverand the cloud server may be used in a quality enforcement process, forupgrading the recognition engine and the software from time to time, forextracting information from the store level to other data users, and soforth. Consistent with another embodiment, the communications betweenthe store server and the cloud server may be discontinuous (purposely orunintentional) and the store server may be configured to operateindependently from the cloud server. For example, the store server maybe configured to generate a record indicative of changes in productplacement that occurred when there was a limited connection (or noconnection) between the store server and the cloud server, and toforward the record to the cloud server once connection is reestablished.

As depicted in FIG. 1, server 135 may be coupled to one or more physicalor virtual storage devices such as database 140. Server 135 may accessdatabase 140 to detect and/or identify products. The detection may occurthrough analysis of features in the image using an algorithm and storeddata. The identification may occur through analysis of product featuresin the image according to stored product models. Consistent with thepresent embodiment, the term “product model” refers to any type ofalgorithm or stored product data that a processor may access or executeto enable the identification of a particular product associated with theproduct model. For example, the product model may include a descriptionof visual and contextual properties of the particular product (e.g., theshape, the size, the colors, the texture, the brand name, the price, thelogo, text appearing on the particular product, the shelf associatedwith the particular product, adjacent products in a planogram, thelocation within the retail store, etc.). In some embodiments, a singleproduct model may be used by server 135 to identify more than one typeof products, such as, when two or more product models are used incombination to enable identification of a product. For example, in somecases, a first product model may be used by server 135 to identify aproduct category (such models may apply to multiple product types, e.g.,shampoo, soft drinks, etc.), and a second product model may be used byserver 135 to identify the product type, product identity, or othercharacteristics associated with a product. In some cases, such productmodels may be applied together (e.g., in series, in parallel, in acascade fashion, in a decision tree fashion, etc.) to reach a productidentification. In other embodiments, a single product model may be usedby server 135 to identify a particular product type (e.g., 6-pack of 16oz Coca-Cola Zero).

Database 140 may be included on a volatile or non-volatile, magnetic,semiconductor, tape, optical, removable, non-removable, or other type ofstorage device or tangible or non-transitory computer-readable medium.Database 140 may also be part of server 135 or separate from server 135.When database 140 is not part of server 135, server 135 may exchangedata with database 140 via a communication link. Database 140 mayinclude one or more memory devices that store data and instructions usedto perform one or more features of the disclosed embodiments. In oneembodiment, database 140 may include any suitable databases, rangingfrom small databases hosted on a work station to large databasesdistributed among data centers. Database 140 may also include anycombination of one or more databases controlled by memory controllerdevices (e.g., server(s), etc.) or software. For example, database 140may include document management systems, Microsoft SQL databases,SharePoint databases, Oracle™ databases, Sybase™ databases, otherrelational databases, or non-relational databases, such as mongo andothers.

Consistent with the present disclosure, image processing unit 130 maycommunicate with output devices 145 to present information derived basedon processing of image data acquired by capturing devices 125. The term“output device” is intended to include all possible types of devicescapable of outputting information from server 135 to users or othercomputer systems (e.g., a display screen, a speaker, a desktop computer,a laptop computer, mobile device, tablet, a PDA, etc.), such as 145A,145B, 145C and 145D. In one embodiment each of the different systemcomponents (i.e., retail stores 105, market research entity 110,suppliers 115, and users 120) may be associated with an output device145, and each system component may be configured to present differentinformation on the output device 145. In one example, server 135 mayanalyze acquired images including representations of shelf spaces. Basedon this analysis, server 135 may compare shelf spaces associated withdifferent products, and output device 145A may present market researchentity 110 with information about the shelf spaces associated withdifferent products. The shelf spaces may also be compared with salesdata, expired products data, and more. Consistent with the presentdisclosure, market research entity 110 may be a part of (or may workwith) supplier 115. In another example, server 135 may determine productcompliance to a predetermined planogram, and output device 145B maypresent to supplier 115 information about the level of productcompliance at one or more retail stores 105 (for example in a specificretail store 105, in a group of retail stores 105 associated withsupplier 115, in all retail stores 105, and so forth). The predeterminedplanogram may be associated with contractual obligations and/or otherpreferences related to the retailer methodology for placement ofproducts on the store shelves. In another example, server 135 maydetermine that a specific store shelf has a type of fault in the productplacement, and output device 145C may present to a manager of retailstore 105 a user-notification that may include information about acorrect display location of a misplaced product, information about astore shelf associated with the misplaced product, information about atype of the misplaced product, and/or a visual depiction of themisplaced product. In another example, server 135 may identify whichproducts are available on the shelf and output device 145D may presentto user 120 an updated list of products.

The components and arrangements shown in FIG. 1 are not intended tolimit the disclosed embodiments, as the system components used toimplement the disclosed processes and features may vary. In oneembodiment, system 100 may include multiple servers 135, and each server135 may host a certain type of service. For example, a first server mayprocess images received from capturing devices 125 to identify at leastsome of the plurality of products in the image, and a second server maydetermine from the identified products in retail stores 105 compliancewith contractual obligations between retail stores 105 and suppliers115. In another embodiment, system 100 may include multiple servers 135,a first type of servers 135 that may process information from specificcapturing devices 125 (e.g., handheld devices of data collection agents)or from specific retail stores 105 (e.g., a server dedicated to aspecific retail store 105 may be placed in or near the store). System100 may further include a second type of servers 135 that collect andprocess information from the first type of servers 135.

FIG. 2 is a block diagram representative of an example configuration ofserver 135. In one embodiment, server 135 may include a bus 200 (or anyother communication mechanism) that interconnects subsystems andcomponents for transferring information within server 135. For example,bus 200 may interconnect a processing device 202, a memory interface204, a network interface 206, and a peripherals interface 208 connectedto an I/O system 210.

Processing device 202, shown in FIG. 2, may include at least oneprocessor configured to execute computer programs, applications,methods, processes, or other software to execute particular instructionsassociated with embodiments described in the present disclosure. Theterm “processing device” refers to any physical device having anelectric circuit that performs a logic operation. For example,processing device 202 may include one or more processors, integratedcircuits, microchips, microcontrollers, microprocessors, all or part ofa central processing unit (CPU), graphics processing unit (GPU), digitalsignal processor (DSP), field programmable gate array (FPGA), or othercircuits suitable for executing instructions or performing logicoperations. Processing device 202 may include at least one processorconfigured to perform functions of the disclosed methods such as amicroprocessor manufactured by Intel™, Nvidia™, manufactured by AMD™,and so forth. Processing device 202 may include a single core ormultiple core processors executing parallel processes simultaneously. Inone example, processing device 202 may be a single core processorconfigured with virtual processing technologies. Processing device 202may implement virtual machine technologies or other technologies toprovide the ability to execute, control, run, manipulate, store, etc.,multiple software processes, applications, programs, etc. In anotherexample, processing device 202 may include a multiple-core processorarrangement (e.g., dual, quad core, etc.) configured to provide parallelprocessing functionalities to allow a device associated with processingdevice 202 to execute multiple processes simultaneously. It isappreciated that other types of processor arrangements could beimplemented to provide the capabilities disclosed herein.

Consistent with the present disclosure, the methods and processesdisclosed herein may be performed by server 135 as a result ofprocessing device 202 executing one or more sequences of one or moreinstructions contained in a non-transitory computer-readable storagemedium. As used herein, a non-transitory computer-readable storagemedium refers to any type of physical memory on which information ordata readable by at least one processor may be stored. Examples includerandom access memory (RAM), read-only memory (ROM), volatile memory,nonvolatile memory, hard drives, CD ROMs, DVDs, flash drives, disks, anyother optical data storage medium, any physical medium with patterns ofholes, a RAM, a PROM, an EPROM, a FLASH-EPROM or any other flash memory,NVRAM, a cache, a register, any other memory chip or cartridge, andnetworked versions of the same. The terms “memory” and“computer-readable storage medium” may refer to multiple structures,such as a plurality of memories or computer-readable storage mediumslocated within server 135, or at a remote location. Additionally, one ormore computer-readable storage mediums may be utilized in implementing acomputer-implemented method. The term “computer-readable storage medium”should be understood to include tangible items and exclude carrier wavesand transient signals.

According to one embodiment, server 135 may include network interface206 (which may also be any communications interface) coupled to bus 200.Network interface 206 may provide one-way or two-way data communicationto a local network, such as network 150. Network interface 206 mayinclude an integrated services digital network (ISDN) card, cable modem,satellite modem, or a modem to provide a data communication connectionto a corresponding type of telephone line. As another example, networkinterface 206 may include a local area network (LAN) card to provide adata communication connection to a compatible LAN. In anotherembodiment, network interface 206 may include an Ethernet port connectedto radio frequency receivers and transmitters and/or optical (e.g.,infrared) receivers and transmitters. The specific design andimplementation of network interface 206 depends on the communicationsnetwork(s) over which server 135 is intended to operate. As describedabove, server 135 may be a cloud server or a local server associatedwith retail store 105. In any such implementation, network interface 206may be configured to send and receive electrical, electromagnetic, oroptical signals, through wires or wirelessly, that may carry analog ordigital data streams representing various types of information. Inanother example, the implementation of network interface 206 may besimilar or identical to the implementation described below for networkinterface 306.

Server 135 may also include peripherals interface 208 coupled to bus200. Peripherals interface 208 may be connected to sensors, devices, andsubsystems to facilitate multiple functionalities. In one embodiment,peripherals interface 208 may be connected to I/O system 210 configuredto receive signals or input from devices and provide signals or outputto one or more devices that allow data to be received and/or transmittedby server 135. In one embodiment I/O system 210 may include or beassociated with output device 145. For example, I/O system 210 mayinclude a touch screen controller 212, an audio controller 214, and/orother input controller(s) 216. Touch screen controller 212 may becoupled to a touch screen 218. Touch screen 218 and touch screencontroller 212 can, for example, detect contact, movement, or breakthereof using any of a plurality of touch sensitivity technologies,including but not limited to capacitive, resistive, infrared, andsurface acoustic wave technologies as well as other proximity sensorarrays or other elements for determining one or more points of contactwith touch screen 218. Touch screen 218 may also, for example, be usedto implement virtual or soft buttons and/or a keyboard. In addition toor instead of touch screen 218, I/O system 210 may include a displayscreen (e.g., CRT, LCD, etc.), virtual reality device, augmented realitydevice, and so forth. Specifically, touch screen controller 212 (ordisplay screen controller) and touch screen 218 (or any of thealternatives mentioned above) may facilitate visual output from server135. Audio controller 214 may be coupled to a microphone 220 and aspeaker 222 to facilitate voice-enabled functions, such as voicerecognition, voice replication, digital recording, and telephonyfunctions. Specifically, audio controller 214 and speaker 222 mayfacilitate audio output from server 135. The other input controller(s)216 may be coupled to other input/control devices 224, such as one ormore buttons, keyboards, rocker switches, thumb-wheel, infrared port,USB port, image sensors, motion sensors, depth sensors, and/or a pointerdevice such as a computer mouse or a stylus.

In some embodiments, processing device 202 may use memory interface 204to access data and a software product stored on a memory device 226.Memory device 226 may include operating system programs for server 135that perform operating system functions when executed by the processingdevice. By way of example, the operating system programs may includeMicrosoft Windows™, Unix™, Linux™, Apple™ operating systems, personaldigital assistant (PDA) type operating systems such as Apple iOS, GoogleAndroid, Blackberry OS, or other types of operating systems.

Memory device 226 may also store communication instructions 228 tofacilitate communicating with one or more additional devices (e.g.,capturing device 125), one or more computers (e.g., output devices145A-145D) and/or one or more servers. Memory device 226 may includegraphical user interface instructions 230 to facilitate graphic userinterface processing; image processing instructions 232 to facilitateimage data processing-related processes and functions; sensor processinginstructions 234 to facilitate sensor-related processing and functions;web browsing instructions 236 to facilitate web browsing-relatedprocesses and functions; and other software instructions 238 tofacilitate other processes and functions. Each of the above identifiedinstructions and applications may correspond to a set of instructionsfor performing one or more functions described above. These instructionsneed not be implemented as separate software programs, procedures, ormodules. Memory device 226 may include additional instructions or fewerinstructions. Furthermore, various functions of server 135 may beimplemented in hardware and/or in software, including in one or moresignal processing and/or application specific integrated circuits. Forexample, server 135 may execute an image processing algorithm toidentify in received images one or more products and/or obstacles, suchas shopping carts, people, and more.

In one embodiment, memory device 226 may store database 140. Database140 may include product type model data 240 (e.g., an imagerepresentation, a list of features, a model obtained by training machinelearning algorithm using training examples, an artificial neuralnetwork, and more) that may be used to identify products in receivedimages; contract-related data 242 (e.g., planograms, promotions data,etc.) that may be used to determine if the placement of products on thestore shelves and/or the promotion execution are consistent withobligations of retail store 105; catalog data 244 (e.g., retail storechain's catalog, retail store's master file, etc.) that may be used tocheck if all product types that should be offered in retail store 105are in fact in the store, if the correct price is displayed next to anidentified product, etc.; inventory data 246 that may be used todetermine if additional products should be ordered from suppliers 115;employee data 248 (e.g., attendance data, records of training provided,evaluation and other performance-related communications, productivityinformation, etc.) that may be used to assign specific store associatesto certain tasks; and calendar data 250 (e.g., holidays, national days,international events, etc.) that may be used to determine if a possiblechange in a product model is associated with a certain event. In otherembodiments of the disclosure, database 140 may store additional typesof data or fewer types of data. Furthermore, various types of data maybe stored in one or more memory devices other than memory device 226.

The components and arrangements shown in FIG. 2 are not intended tolimit the disclosed embodiments. As will be appreciated by a personskilled in the art having the benefit of this disclosure, numerousvariations and/or modifications may be made to the depictedconfiguration of server 135. For example, not all components may beessential for the operation of server 135 in all cases. Any componentmay be located in any appropriate part of server 135, and the componentsmay be rearranged into a variety of configurations while providing thefunctionality of the disclosed embodiments. For example, some serversmay not include some of the elements shown in I/O system 215.

FIG. 3 is a block diagram representation of an example configuration ofcapturing device 125. In one embodiment, capturing device 125 mayinclude a processing device 302, a memory interface 304, a networkinterface 306, and a peripherals interface 308 connected to image sensor310. These components may be separated or may be integrated in one ormore integrated circuits. The various components in capturing device 125may be coupled by one or more communication buses or signal lines (e.g.,bus 300). Different aspects of the functionalities of the variouscomponents in capturing device 125 may be understood from thedescription above regarding components of server 135 having similarfunctionality.

According to one embodiment, network interface 306 may be used tofacilitate communication with server 135. Network interface 306 may bean Ethernet port connected to radio frequency receivers and transmittersand/or optical receivers and transmitters. The specific design andimplementation of network interface 306 depends on the communicationsnetwork(s) over which capturing device 125 is intended to operate. Forexample, in some embodiments, capturing device 125 may include a networkinterface 306 designed to operate over a GSM network, a GPRS network, anEDGE network, a Wi-Fi or WiMax network, a Bluetooth® network, etc. Inanother example, the implementation of network interface 306 may besimilar or identical to the implementation described above for networkinterface 206.

In the example illustrated in FIG. 3, peripherals interface 308 ofcapturing device 125 may be connected to at least one image sensor 310associated with at least one lens 312 for capturing image data in anassociated field of view. In some configurations, capturing device 125may include a plurality of image sensors associated with a plurality oflenses 312. In other configurations, image sensor 310 may be part of acamera included in capturing device 125. According to some embodiments,peripherals interface 308 may also be connected to other sensors (notshown), such as a motion sensor, a light sensor, infrared sensor, soundsensor, a proximity sensor, a temperature sensor, a biometric sensor, orother sensing devices to facilitate related functionalities. Inaddition, a positioning sensor may also be integrated with, or connectedto, capturing device 125. For example, such positioning sensor may beimplemented using one of the following technologies: Global PositioningSystem (GPS), GLObal NAvigation Satellite System (GLONASS), Galileoglobal navigation system, BeiDou navigation system, other GlobalNavigation Satellite Systems (GNSS), Indian Regional NavigationSatellite System (IRNSS), Local Positioning Systems (LPS), Real-TimeLocation Systems (RTLS), Indoor Positioning System (IPS), Wi-Fi basedpositioning systems, cellular triangulation, and so forth. For example,the positioning sensor may be built into mobile capturing device 125,such as smartphone devices. In another example, position software mayallow mobile capturing devices to use internal or external positioningsensors (e.g., connecting via a serial port or Bluetooth).

Consistent with the present disclosure, capturing device 125 may includedigital components that collect data from image sensor 310, transform itinto an image, and store the image on a memory device 314 and/ortransmit the image using network interface 306. In one embodiment,capturing device 125 may be fixedly mountable to a store shelf or toother objects in the retail store (such as walls, ceilings, floors,refrigerators, checkout stations, displays, dispensers, rods which maybe connected to other objects in the retail store, and so forth). In oneembodiment, capturing device 125 may be split into at least two housingssuch that only image sensor 310 and lens 312 may be visible on the storeshelf, and the rest of the digital components may be located in aseparate housing. An example of this type of capturing device isdescribed below with reference to FIGS. 5-7.

Consistent with the present disclosure, capturing device 125 may usememory interface 304 to access memory device 314. Memory device 314 mayinclude high-speed, random access memory and/or non-volatile memory suchas one or more magnetic disk storage devices, one or more opticalstorage devices, and/or flash memory (e.g., NAND, NOR) to store capturedimage data. Memory device 314 may store operating system instructions316, such as DARWIN, RTXC, LINUX, iOS, UNIX, LINUX, OS X, WINDOWS, or anembedded operating system such as VXWorkS. Operating system 316 mayinclude instructions for handling basic system services and forperforming hardware dependent tasks. In some implementations, operatingsystem 316 may include a kernel (e.g., UNIX kernel, LINUX kernel, etc.).In addition, memory device 314 may store capturing instructions 318 tofacilitate processes and functions related to image sensor 310;graphical user interface instructions 320 that enables a user associatedwith capturing device 125 to control the capturing device and/or toacquire images of an area-of-interest in a retail establishment; andapplication instructions 322 to facilitate a process for monitoringcompliance of product placement or other processes.

The components and arrangements shown in FIG. 3 are not intended tolimit the disclosed embodiments. As will be appreciated by a personskilled in the art having the benefit of this disclosure, numerousvariations and/or modifications may be made to the depictedconfiguration of capturing device 125. For example, not all componentsare essential for the operation of capturing device 125 in all cases.Any component may be located in any appropriate part of capturing device125, and the components may be rearranged into a variety ofconfigurations while providing the functionality of the disclosedembodiments. For example, some capturing devices may not have lenses,and other capturing devices may include an external memory deviceinstead of memory device 314.

In some embodiments, machine learning algorithms (also referred to asmachine learning models in the present disclosure) may be trained usingtraining examples, for example in the cases described below. Somenon-limiting examples of such machine learning algorithms may includeclassification algorithms, data regressions algorithms, imagesegmentation algorithms, visual detection algorithms (such as objectdetectors, face detectors, person detectors, motion detectors, edgedetectors, etc.), visual recognition algorithms (such as facerecognition, person recognition, object recognition, etc.), speechrecognition algorithms, mathematical embedding algorithms, naturallanguage processing algorithms, support vector machines, random forests,nearest neighbors algorithms, deep learning algorithms, artificialneural network algorithms, convolutional neural network algorithms,recurrent neural network algorithms, linear machine learning models,non-linear machine learning models, ensemble algorithms, and so forth.For example, a trained machine learning algorithm may comprise aninference model, such as a predictive model, a classification model, adata regression model, a clustering model, a segmentation model, anartificial neural network (such as a deep neural network, aconvolutional neural network, a recurrent neural network, etc.), arandom forest, a support vector machine, and so forth. In some examples,the training examples may include example inputs together with thedesired outputs corresponding to the example inputs. Further, in someexamples, training machine learning algorithms using the trainingexamples may generate a trained machine learning algorithm, and thetrained machine learning algorithm may be used to estimate outputs forinputs not included in the training examples. In some examples,engineers, scientists, processes and machines that train machinelearning algorithms may further use validation examples and/or testexamples. For example, validation examples and/or test examples mayinclude example inputs together with the desired outputs correspondingto the example inputs, a trained machine learning algorithm and/or anintermediately trained machine learning algorithm may be used toestimate outputs for the example inputs of the validation examplesand/or test examples, the estimated outputs may be compared to thecorresponding desired outputs, and the trained machine learningalgorithm and/or the intermediately trained machine learning algorithmmay be evaluated based on a result of the comparison. In some examples,a machine learning algorithm may have parameters and hyper parameters,where the hyper parameters may be set manually by a person orautomatically by a process external to the machine learning algorithm(such as a hyper parameter search algorithm), and the parameters of themachine learning algorithm may be set by the machine learning algorithmbased on the training examples. In some implementations, thehyper-parameters may be set based on the training examples and thevalidation examples, and the parameters may be set based on the trainingexamples and the selected hyper-parameters. For example, given thehyper-parameters, the parameters may be conditionally independent of thevalidation examples.

In some embodiments, trained machine learning algorithms (also referredto as machine learning models and trained machine learning models in thepresent disclosure) may be used to analyze inputs and generate outputs,for example in the cases described below. In some examples, a trainedmachine learning algorithm may be used as an inference model that whenprovided with an input generates an inferred output. For example, atrained machine learning algorithm may include a classificationalgorithm, the input may include a sample, and the inferred output mayinclude a classification of the sample (such as an inferred label, aninferred tag, and so forth). In another example, a trained machinelearning algorithm may include a regression model, the input may includea sample, and the inferred output may include an inferred valuecorresponding to the sample. In yet another example, a trained machinelearning algorithm may include a clustering model, the input may includea sample, and the inferred output may include an assignment of thesample to at least one cluster. In an additional example, a trainedmachine learning algorithm may include a classification algorithm, theinput may include an image, and the inferred output may include aclassification of an item depicted in the image. In yet another example,a trained machine learning algorithm may include a regression model, theinput may include an image, and the inferred output may include aninferred value corresponding to an item depicted in the image (such asan estimated property of the item, such as size, volume, age of a persondepicted in the image, cost of a product depicted in the image, and soforth). In an additional example, a trained machine learning algorithmmay include an image segmentation model, the input may include an image,and the inferred output may include a segmentation of the image. In yetanother example, a trained machine learning algorithm may include anobject detector, the input may include an image, and the inferred outputmay include one or more detected objects in the image and/or one or morelocations of objects within the image. In some examples, the trainedmachine learning algorithm may include one or more formulas and/or oneor more functions and/or one or more rules and/or one or moreprocedures, the input may be used as input to the formulas and/orfunctions and/or rules and/or procedures, and the inferred output may bebased on the outputs of the formulas and/or functions and/or rulesand/or procedures (for example, selecting one of the outputs of theformulas and/or functions and/or rules and/or procedures, using astatistical measure of the outputs of the formulas and/or functionsand/or rules and/or procedures, and so forth).

In some embodiments, artificial neural networks may be configured toanalyze inputs and generate corresponding outputs, for example in thecases described below. Some non-limiting examples of such artificialneural networks may comprise shallow artificial neural networks, deepartificial neural networks, feedback artificial neural networks, feedforward artificial neural networks, autoencoder artificial neuralnetworks, probabilistic artificial neural networks, time delayartificial neural networks, convolutional artificial neural networks,recurrent artificial neural networks, long short term memory artificialneural networks, and so forth. In some examples, an artificial neuralnetwork may be configured manually. For example, a structure of theartificial neural network may be selected manually, a type of anartificial neuron of the artificial neural network may be selectedmanually, a parameter of the artificial neural network (such as aparameter of an artificial neuron of the artificial neural network) maybe selected manually, and so forth. In some examples, an artificialneural network may be configured using a machine learning algorithm. Forexample, a user may select hyper-parameters for the artificial neuralnetwork and/or the machine learning algorithm, and the machine learningalgorithm may use the hyper-parameters and training examples todetermine the parameters of the artificial neural network, for exampleusing back propagation, using gradient descent, using stochasticgradient descent, using mini-batch gradient descent, and so forth. Insome examples, an artificial neural network may be created from two ormore other artificial neural networks by combining the two or more otherartificial neural networks into a single artificial neural network.

Some non-limiting examples of image data may include images, grayscaleimages, color images, 2D images, 3D images, videos, 2D videos, 3Dvideos, frames, footages, data derived from other image data, and soforth. In some embodiments, analyzing image data (for example by themethods, steps and modules described herein) may comprise analyzing theimage data to obtain a preprocessed image data, and subsequentlyanalyzing the image data and/or the preprocessed image data to obtainthe desired outcome. One of ordinary skill in the art will recognizethat the following embodiments are provided by way of example, and thatthe image data may be preprocessed using other kinds of preprocessingmethods. In some examples, the image data may be preprocessed bytransforming the image data using a transformation function to obtain atransformed image data, and the preprocessed image data may comprise thetransformed image data. For example, the transformed image data maycomprise one or more convolutions of the image data. For example, thetransformation function may comprise one or more image filters, such aslow-pass filters, high-pass filters, band-pass filters, all-passfilters, and so forth. In some examples, the transformation function maycomprise a nonlinear function. In some examples, the image data may bepreprocessed by smoothing at least parts of the image data, for exampleusing Gaussian convolution, using a median filter, and so forth. In someexamples, the image data may be preprocessed to obtain a differentrepresentation of the image data. For example, the preprocessed imagedata may comprise: a representation of at least part of the image datain a frequency domain; a Discrete Fourier Transform of at least part ofthe image data; a Discrete Wavelet Transform of at least part of theimage data; a time/frequency representation of at least part of theimage data; a representation of at least part of the image data in alower dimension; a lossy representation of at least part of the imagedata; a lossless representation of at least part of the image data; atime ordered series of any of the above; any combination of the above;and so forth. In some examples, the image data may be preprocessed toextract edges, and the preprocessed image data may comprise informationbased on and/or related to the extracted edges. In some examples, theimage data may be preprocessed to extract image features from the imagedata. Some non-limiting examples of such image features may compriseinformation based on and/or related to: edges; corners; blobs; ridges;Scale Invariant Feature Transform (SIFT) features; temporal features;and so forth. In some examples, analyzing the image data may includecalculating at least one convolution of at least a portion of the imagedata, and using the calculated at least one convolution to calculate atleast one resulting value and/or to make determinations,identifications, recognitions, classifications, and so forth.

In some embodiments, analyzing image data (for example by the methods,steps and modules described herein) may comprise analyzing the imagedata and/or the preprocessed image data using one or more rules,functions, procedures, artificial neural networks, object detectionalgorithms, face detection algorithms, visual event detectionalgorithms, action detection algorithms, motion detection algorithms,background subtraction algorithms, inference models, and so forth. Somenon-limiting examples of such inference models may include: an inferencemodel preprogrammed manually; a classification model; a regressionmodel; a result of training algorithms, such as machine learningalgorithms and/or deep learning algorithms, on training examples, wherethe training examples may include examples of data instances, and insome cases, a data instance may be labeled with a corresponding desiredlabel and/or result; and so forth. In some embodiments, analyzing imagedata (for example by the methods, steps and modules described herein)may comprise analyzing pixels, voxels, point cloud, range data, etc.included in the image data.

FIGS. 4A-4C illustrate example configurations for capturing image datain retail store 105 according to disclosed embodiments. FIG. 4Aillustrates how an aisle 400 of retail store 105 may be imaged using aplurality of capturing devices 125 fixedly connected to store shelves.FIG. 4B illustrates how aisle 400 of retail store 105 may be imagedusing a handheld communication device. FIG. 4C illustrates how aisle 400of retail store 105 may be imaged by robotic devices equipped withcameras.

With reference to FIG. 4A and consistent with the present disclosure,retail store 105 may include a plurality of capturing devices 125fixedly mounted (for example, to store shelves, walls, ceilings, floors,refrigerators, checkout stations, displays, dispensers, rods which maybe connected to other objects in the retail store, and so forth) andconfigured to collect image data. As depicted, one side of an aisle 400may include a plurality of capturing devices 125 (e.g., 125A, 125B, and125C) fixedly mounted thereon and directed such that they may captureimages of an opposing side of aisle 400. The plurality of capturingdevices 125 may be connected to an associated mobile power source (e.g.,one or more batteries), to an external power supply (e.g., a powergrid), obtain electrical power from a wireless power transmissionsystem, and so forth. As depicted in FIG. 4A, the plurality of capturingdevices 125 may be placed at different heights and at least theirvertical fields of view may be adjustable. Generally, both sides ofaisle 400 may include capturing devices 125 in order to cover both sidesof aisle 400.

Differing numbers of capturing devices 125 may be used to cover shelvingunit 402. In addition, there may be an overlap region in the horizontalfield of views of some of capturing devices 125. For example, thehorizontal fields of view of capturing devices (e.g., adjacent capturingdevices) may at least partially overlap with one another. In anotherexample, one capturing device may have a lower field of view than thefield of view of a second capturing device, and the two capturingdevices may have at least partially overlapping fields of view.According to one embodiment, each capturing device 125 may be equippedwith network interface 306 for communicating with server 135. In oneembodiment, the plurality of capturing devices 125 in retail store 105may be connected to server 135 via a single WLAN. Network interface 306may transmit information associated with a plurality of images capturedby the plurality of capturing devices 125 for analysis purposes. In oneexample, server 135 may determine an existence of an occlusion event(such as, by a person, by store equipment, such as a ladder, cart, etc.)and may provide a notification to resolve the occlusion event. Inanother example, server 135 may determine if a disparity exists betweenat least one contractual obligation and product placement as determinedbased on automatic analysis of the plurality of images. The transmittedinformation may include raw images, cropped images, processed imagedata, data about products identified in the images, and so forth.Network interface 306 may also transmit information identifying thelocation of the plurality capturing devices 125 in retail store 105.

With reference to FIG. 4B and consistent with the present disclosure,server 135 may receive image data captured by users 120. In a firstembodiment, server 135 may receive image data acquired by storeassociates. In one implementation, a handheld device of a storeassociate (e.g., capturing device 125D) may display a real-time videostream captured by the image sensor of the handheld device. Thereal-time video stream may be augmented with markings identifying to thestore associate an area-of-interest that needs manual capturing ofimages. One of the situations in which manual image capture may bedesirable may occur where the area-of-interest is outside the fields ofview of a plurality of cameras fixedly connected to store shelves inaisle 400. In other situations, manual capturing of images of anarea-of-interest may be desirable when a current set of acquired imagesis out of date (e.g., obsolete in at least one respect) or of poorquality (e.g., lacking focus, obstacles, lesser resolution, lack oflight, etc.). Additional details of this embodiment are described inApplicant's International Patent Application No. PCT/IB2018/001107,which is incorporated herein by reference.

In a second embodiment, server 135 may receive image data acquired bycrowd sourcing. In one exemplary implementation, server 135 may providea request to a detected mobile device for an updated image of thearea-of-interest in aisle 400. The request may include an incentive(e.g., $2 discount) to user 120 for acquiring the image. In response tothe request, user 120 may acquire and transmit an up-to-date image ofthe area-of-interest. After receiving the image from user 120, server135 may transmit the accepted incentive or agreed upon reward to user120. The incentive may comprise a text notification and a redeemablecoupon. In some embodiments, the incentive may include a redeemablecoupon for a product associated with the area-of-interest. Server 135may generate image-related data based on aggregation of data from imagesreceived from crowd sourcing and from images received from a pluralityof cameras fixedly connected to store shelves. Additional details ofthis embodiment are described in Applicant's International PatentApplication No. PCT/IB2017/000919, which is incorporated herein byreference.

With reference to FIG. 4C and consistent with the present disclosure,server 135 may receive image data captured by robotic devices withcameras traversing in aisle 400. The present disclosure is not limitedto the type of robotic devices used to capture images of retail store105. In some embodiments, the robotic devices may include a robot on atrack (e.g., a Cartesian robot configured to move along an edge of ashelf or in parallel to a shelf, such as capturing device 125E), a drone(e.g., capturing device 125F), and/or a robot that may move on the floorof the retail store (e.g., a wheeled robot such as capturing device125G, a legged robot, a snake-like robot, etc.). The robotic devices maybe controlled by server 135 and may be operated remotely orautonomously. In one example, server 135 may instruct capturing device125E to perform periodic scans at times when no customers or otherobstructions are identified in aisle 400. Specifically, capturing device125E may be configured to move along store shelf 404 and to captureimages of products placed on store shelf 404, products placed on storeshelf 406, or products located on shelves opposite store shelf (e.g.,store shelf 408). In another example, server 135 may instruct capturingdevice 125F to perform a scan of all the area of retail store 105 beforethe opening hour. In another example, server 135 may instruct capturingdevice 125G to capture a specific area-of-interest, similar as describedabove with reference to receiving images acquired by the storeassociates. In some embodiments, robotic capturing devices (such as 125Fand 125G) may include an internal processing unit that may allow them tonavigate autonomously within retail store 105. For example, the roboticcapturing devices may use input from sensors (e.g., image sensors, depthsensors, proximity sensors, etc.), to avoid collision with objects orpeople, and to complete the scan of the desired area of retail store105.

As discussed above with reference to FIG. 4A, the image datarepresentative of products displayed on store shelves may be acquired bya plurality of stationary capturing devices 125 fixedly mounted in theretail store. One advantage of having stationary image capturing devicesspread throughout retail store 105 is the potential for acquiringproduct images from set locations and on an ongoing basis such thatup-to-date product status may be determined for products throughout aretail store at any desired periodicity (e.g., in contrast to a movingcamera system that may acquire product images more infrequently).However, there may be certain challenges in this approach. The distancesand angles of the image capturing devices relative to the capturedproducts should be selected such as to enable adequate productidentification, especially when considered in view of image sensorresolution and/or optics specifications. For example, a capturing deviceplaced on the ceiling of retail store 105 may have sufficientresolutions and optics to enable identification of large products (e.g.,a pack of toilet paper), but may be insufficient for identifying smallerproducts (e.g., deodorant packages). The image capturing devices shouldnot occupy shelf space that is reserved for products for sale. The imagecapturing devices should not be positioned in places where there is alikelihood that their fields of view will be regularly blocked bydifferent objects. The image capturing devices should be able tofunction for long periods of time with minimum maintenance. For example,a requirement for frequent replacement of batteries may render certainimage acquisition systems cumbersome to use, especially where many imageacquisition devices are in use throughout multiple locations in a retailstore and across multiple retail stores. The image capturing devicesshould also include processing capabilities and transmissioncapabilities for providing real time or near real time image data aboutproducts. The disclosed image acquisition systems address thesechallenges.

FIG. 5A illustrates an example of a system 500 for acquiring images ofproducts in retail store 105. Throughout the disclosure, capturingdevice 125 may refer to a system, such as system 500 shown in FIG. 5A.As shown, system 500 may include a first housing 502 configured forlocation on a retail shelving unit (e.g., as illustrated in FIG. 5B),and a second housing 504 configured for location on the retail shelvingunit separate from first housing 502. The first and the second housingmay be configured for mounting on the retail shelving unit in anysuitable way (e.g., screws, bolts, clamps, adhesives, magnets,mechanical means, chemical means, etc.). In some embodiments, firsthousing 502 may include an image capture device 506 (e.g., a cameramodule that may include image sensor 310) and second housing 504 mayinclude at least one processor (e.g., processing device 302) configuredto control image capture device 506 and also to control a networkinterface (e.g., network interface 306) for communicating with a remoteserver (e.g., server 135).

System 500 may also include a data conduit 508 extending between firsthousing 502 and second housing 504. Data conduit 508 may be configuredto enable transfer of control signals from the at least one processor toimage capture device 506 and to enable collection of image data acquiredby image capture device 506 for transmission by the network interface.Consistent with the present disclosure, the term “data conduit” mayrefer to a communications channel that may include either a physicaltransmission medium such as a wire or a logical connection over amultiplexed medium such as a radio channel. In some embodiments, dataconduit 508 may be used for conveying image data from image capturedevice 506 to at least one processor located in second housing 504.Consistent with one implementation of system 500, data conduit 508 mayinclude flexible printed circuits and may have a length of at leastabout 5 cm, at least about 10 cm, at least about 15 cm, etc. The lengthof data conduit 508 may be adjustable to enable placement of firsthousing 502 separately from second housing 504. For example, in someembodiments, data conduit may be retractable within second housing 504such that the length of data conduit exposed between first housing 502and second housing 504 may be selectively adjusted.

In one embodiment, the length of data conduit 508 may enable firsthousing 502 to be mounted on a first side of a horizontal store shelffacing the aisle (e.g., store shelf 510 illustrated in FIG. 5B) andsecond housing 504 to be mounted on a second side of store shelf 510that faces the direction of the ground (e.g., an underside of a storeshelf). In this embodiment, data conduit 508 may be configured to bendaround an edge of store shelf 510 or otherwise adhere/follow contours ofthe shelving unit. For example, a first portion of data conduit 508 maybe configured for location on the first side of store shelf 510 (e.g., aside facing an opposing retail shelving unit across an aisle) and asecond portion of data conduit 508 may be configured for location on asecond side of store shelf 510 (e.g., an underside of the shelf, whichin some cases may be orthogonal to the first side). The second portionof data conduit 508 may be longer than the first portion of data conduit508. Consistent with another embodiment, data conduit 508 may beconfigured for location within an envelope of a store shelf. Forexample, the envelope may include the outer boundaries of a channellocated within a store shelf, a region on an underside of an L-shapedstore shelf, a region between two store shelves, etc. Consistent withanother implementation of system 500 discussed below, data conduit 508may include a virtual conduit associated with a wireless communicationslink between first housing 502 and second housing 504.

FIG. 5B illustrates an exemplary configuration for mounting firsthousing 502 on store shelf 510. Consistent with the present disclosure,first housing 502 may be placed on store shelf 510, next to or embeddedin a plastic cover that may be used for displaying prices.Alternatively, first housing 502 may be placed or mounted on any otherlocation in retail store 105. For example, first housing 502 may beplaced or mounted on the walls, on the ceiling, on refrigerator units,on display units, and more. The location and/or orientation of firsthousing 502 may be selected such that a field of view of image capturedevice 506 may cover at least a portion of an opposing retail shelvingunit. Consistent with the present disclosure, image capture device 506may have a view angle of between 50 and 80 degrees, about 62 degrees,about 67 degrees, or about 75 degrees. Consistent with the presentdisclosure, image capture device 506 may include an image sensor havingsufficient image resolution to enable detection of text associated withlabels on an opposing retail shelving unit. In one embodiment, the imagesensor may include m*n pixels. For example, image capture device 506 mayhave an 8 MP image sensor that includes an array of 3280*2464 pixels.Each pixel may include at least one photo-voltaic cell that converts thephotons of the incident light to an electric signal. The electricalsignal may be converted to digital data by an A/D converter andprocessed by the image processor (ISP). In one embodiment, the imagesensor of image capture device 506 may be associated with a pixel sizeof between 1.1×1.1 um² and 1.7×1.7 um², for example, 1.4×1.4 um².

Consistent with the present disclosure, image capture device 506 may beassociated with a lens (e.g., lens 312) having a fixed focal lengthselected according to a distance expected to be encountered betweenretail shelving units on opposite sides of an aisle (e.g., distance d1shown in FIG. 6A) and/or according to a distance expected to beencountered between a side of a shelving unit facing the aisle on oneside of an aisle and a side of a shelving unit facing away of the aisleon the other side of the aisle (e.g., distance d2 shown in FIG. 6A). Thefocal length may also be based on any other expected distance betweenthe image acquisition device and products to be imaged. As used herein,the term “focal length” refers to the distance from the optical centerof the lens to a point where objects located at the point aresubstantially brought into focus. In contrast to zoom lenses, in fixedlenses the focus is not adjustable. The focus is typically set at thetime of lens design and remains fixed. In one embodiment, the focallength of lens 312 may be selected based on the distance between twosides of aisles in the retail store (e.g., distance d1, distance d2, andso forth). In some embodiments, image capture device 506 may include alens with a fixed focal length having a fixed value between 2.5 mm and4.5 mm, such as about 3.1 mm, about 3.4 mm, about 3.7 mm. For example,when distance d1 between two opposing retail shelving units is about 2meters, the focal length of the lens may be about 3.6 mm. Unlessindicated otherwise, the term “about” with regards to a numeric value isdefined as a variance of up to 5% with respect to the stated value. Ofcourse, image capture devices having non-fixed focal lengths may also beused depending on the requirements of certain imaging environments, thepower and space resources available, etc.

FIG. 5C illustrates an exploded view of second housing 504. In someembodiments, the network interface located in second housing 504 (e.g.,network interface 306) may be configured to transmit to server 135information associated with a plurality of images captured by imagecapture device 506. For example, the transmitted information may be usedto determine if a disparity exists between at least one contractualobligation (e.g. planogram) and product placement. In one example, thenetwork interface may support transmission speeds of 0.5 Mb/s, 1 Mb/s, 5Mb/s, or more. Consistent with the present disclosure, the networkinterface may allow different modes of operations to be selected, suchas: high-speed, slope-control, or standby. In high-speed mode,associated output drivers may have fast output rise and fall times tosupport high-speed bus rates; in slope-control, the electromagneticinterference may be reduced and the slope (i.e., the change of voltageper unit of time) may be proportional to the current output; and instandby mode, the transmitter may be switched off and the receiver mayoperate at a lower current.

Consistent with the present disclosure, second housing 504 may include apower port 512 for conveying energy from a power source to first housing502. In one embodiment, second housing 504 may include a section for atleast one mobile power source 514 (e.g., in the depicted configurationthe section is configured to house four batteries). The at least onemobile power source may provide sufficient power to enable image capturedevice 506 to acquire more than 1,000 pictures, more than 5,000pictures, more than 10,000 pictures, or more than 15,000 pictures, andto transmit them to server 135. In one embodiment, mobile power source514 located in a single second housing 504 may power two or more imagecapture devices 506 mounted on the store shelf. For example, as depictedin FIGS. 6A and 6B, a single second housing 504 may be connected to aplurality of first housings 502 with a plurality of image capturedevices 506 covering different (overlapping or non-overlapping) fieldsof view. Accordingly, the two or more image capture devices 506 may bepowered by a single mobile power source 514 and/or the data captured bytwo or more image capture devices 506 may be processed to generate apanoramic image by a single processing device located in second housing504. In addition to mobile power source 514 or as an alternative tomobile power source 514, second housing 504 may also be connected to anexternal power source. For example, second housing 504 may be mounted toa store shelf and connected to an electric power grid. In this example,power port 512 may be connected to the store shelf through a wire forproviding electrical power to image capture device 506. In anotherexample, a retail shelving unit or retail store 105 may include awireless power transmission system, and power port 512 may be connectedto a device configured to obtain electrical power from the wirelesspower transmission system. In addition, as discussed below, system 500may use power management policies to reduce the power consumption. Forexample, system 500 may use selective image capturing and/or selectivetransmission of images to reduce the power consumption or conservepower.

FIG. 6A illustrates a schematic diagram of a top view of aisle 600 inretail store 105 with multiple image acquisition systems 500 (e.g.,500A, 500B, 500C, 500D, and 500E) deployed thereon for acquiring imagesof products. Aisle 600 may include a first retail shelving unit 602 anda second retail shelving unit 604 that opposes first retail shelvingunit 602. In some embodiments, different numbers of systems 500 may bemounted on opposing retail shelving units. For example, system 500A(including first housing 502A, second housing 504A, and data conduit508A), system 500B (including first housing 502B second housing 504B,and data conduit 508B), and system 500C (including first housing 502C,second housing 504C, and data conduit 508C) may be mounted on firstretail shelving unit 602; and system 500D (including first housing502D1, first housing 502D2, second housing 504D, and data conduits 508D1and 508D2) and system 500E (including first housing 502E1, first housing502E2, second housing 504E, and data conduits 508E1 and 508E2) may bemounted on second retail shelving unit 604. Consistent with the presentdisclosure, image capture device 506 may be configured relative to firsthousing 502 such that an optical axis of image capture device 506 isdirected toward an opposing retail shelving unit when first housing 502is fixedly mounted on a retail shelving unit. For example, optical axis606 of the image capture device associated with first housing 502B maybe directed towards second retail shelving unit 604 when first housing502B is fixedly mounted on first retail shelving unit 602. A singleretail shelving unit may hold a number of systems 500 that include aplurality of image capturing devices. Each of the image capturingdevices may be associated with a different field of view directed towardthe opposing retail shelving unit. Different vantage points ofdifferently located image capture devices may enable image acquisitionrelative to different sections of a retail shelf. For example, at leastsome of the plurality of image capturing devices may be fixedly mountedon shelves at different heights. Examples of such a deployment areillustrated in FIGS. 4A and 6B.

As shown in FIG. 6A each first housing 502 may be associated with a dataconduit 508 that enables exchanging of information (e.g., image data,control signals, etc.) between the at least one processor located insecond housing 504 and image capture device 506 located in first housing502. In some embodiments, data conduit 508 may include a wiredconnection that supports data-transfer and may be used to power imagecapture device 506 (e.g., data conduit 508A, data conduit 508B, dataconduit 508D1, data conduit 508D2, data conduit 508E1, and data conduit508E2). Consistent with these embodiments, data conduit 508 may complywith a wired standard such as USB, Micro-USB, HDMI, Micro-HDMI,Firewire, Apple, etc. In other embodiments, data conduit 508 may be awireless connection, such as a dedicated communications channel betweenthe at least one processor located in second housing 504 and imagecapture device 506 located in first housing 502 (e.g., data conduit508C). In one example, the communications channel may be established bytwo Near Field Communication (NFC) transceivers. In other examples,first housing 502 and second housing 504 may include interface circuitsthat comply with other short-range wireless standards such as Bluetooth,WiFi, ZigBee, etc.

In some embodiments of the disclosure, the at least one processor ofsystem 500 may cause at least one image capture device 506 toperiodically capture images of products located on an opposing retailshelving unit (e.g., images of products located on a shelf across anaisle from the shelf on which first housing 502 is mounted). The term“periodically capturing images” includes capturing an image or images atpredetermined time intervals (e.g., every minute, every 30 minutes,every 150 minutes, every 300 minutes, etc.), capturing video, capturingan image every time a status request is received, and/or capturing animage subsequent to receiving input from an additional sensor, forexample, an associated proximity sensor. Images may also be capturedbased on various other triggers or in response to various other detectedevents. In some embodiments, system 500 may receive an output signalfrom at least one sensor located on an opposing retail shelving unit.For example, system 500B may receive output signals from a sensingsystem located on second retail shelving unit 604. The output signalsmay be indicative of a sensed lifting of a product from second retailshelving unit 604 or a sensed positioning of a product on second retailshelving unit 604. In response to receiving the output signal from theat least one sensor located on second retail shelving unit 604, system500B may cause image capture device 506 to capture one or more images ofsecond retail shelving unit 604. Additional details on a sensing system,including the at least one sensor that generates output signalsindicative of a sensed lifting of a product from an opposing retailshelving unit, is discussed below with reference to FIGS. 8-10.

Consistent with embodiments of the disclosure, system 500 may detect anobject 608 in a selected area between first retail shelving unit 602 andsecond retail shelving unit 604. Such detection may be based on theoutput of one or more dedicated sensors (e.g., motion detectors, etc.)and/or may be based on image analysis of one or more images acquired byan image acquisition device. Such images, for example, may include arepresentation of a person or other object recognizable through variousimage analysis techniques (e.g., trained neural networks, Fouriertransform analysis, edge detection, filters, face recognition, etc.).The selected area may be associated with distance d1 between firstretail shelving unit 602 and second retail shelving unit 604. Theselected area may be within the field of view of image capture device506 or an area where the object causes an occlusion of a region ofinterest (such as a shelf, a portion of a shelf being monitored, andmore). Upon detecting object 608, system 500 may cause image capturedevice 506 to forgo image acquisition while object 608 is within theselected area. In one example, object 608 may be an individual, such asa customer or a store associate. In another example, detected object 608may be an inanimate object, such as a cart, box, carton, one or moreproducts, cleaning robots, etc. In the example illustrated in FIG. 6A,system 500A may detect that object 608 has entered into its associatedfield of view (e.g., using a proximity sensor) and may instruct imagecapturing device 506 to forgo image acquisition. In alternativeembodiments, system 500 may analyze a plurality of images acquired byimage capture device 506 and identify at least one image of theplurality of images that includes a representation of object 608.Thereafter, system 500 may avoid transmission of at least part of the atleast one identified image and/or information based on the at least oneidentified image to server 135.

As shown in FIG. 6A, the at least one processor contained in a secondhousing 504 may control a plurality of image capture devices 506contained in a plurality of first housings 502 (e.g., systems 500D and500E). Controlling image capturing device 506 may include instructingimage capturing device 506 to capture an image and/or transmit capturedimages to a remote server (e.g., server 135). In some cases, each of theplurality of image capture devices 506 may have a field of view that atleast partially overlaps with a field of view of at least one otherimage capture device 506 from among plurality of image capture devices506. In one embodiment, the plurality of image capture devices 506 maybe configured for location on one or more horizontal shelves and may bedirected to substantially different areas of the opposing first retailshelving unit. In this embodiment, the at least one processor maycontrol the plurality of image capture devices such that each of theplurality of image capture devices may capture an image at a differenttime. For example, system 500E may have a second housing 504E with atleast one processor that may instruct a first image capturing devicecontained in first housing 502E1 to capture an image at a first time andmay instruct a second image capturing device contained in first housing502E2 to capture an image at a second time which differs from the firsttime. Capturing images in different times (or forwarding them to the atleast one processor at different times) may assist in processing theimages and writing the images in the memory associated with the at leastone processor.

FIG. 6B illustrates a perspective view assembly diagram depicting aportion of a retail shelving unit 620 with multiple systems 500 (e.g.,500F, 500G, 500H, 500I, and 500J) deployed thereon for acquiring imagesof products. Retail shelving unit 620 may include horizontal shelves atdifferent heights. For example, horizontal shelves 622A, 622B, and 622Care located below horizontal shelves 622D, 622E, and 622F. In someembodiments, a different number of systems 500 may be mounted on shelvesat different heights. For example, system 500F (including first housing502F and second housing 504F), system 500G (including first housing 502Gand second housing 504G), and system 500H (including first housing 502Hand second housing 504H) may be mounted on horizontal shelves associatedwith a first height; and system 500I (including first housing 502I,second housing 504I, and a projector 632) and system 500J (includingfirst housing 502J1, first housing 502J2, and second housing 504J) maybe mounted on horizontal shelves associated with a second height. Insome embodiments, retail shelving unit 620 may include a horizontalshelf with at least one designated place (not shown) for mounting ahousing of image capturing device 506. The at least one designated placemay be associated with connectors such that first housing 502 may befixedly mounted on a side of horizontal shelf 622 facing an opposingretail shelving unit using the connectors.

Consistent with the present disclosure, system 500 may be mounted on aretail shelving unit that includes at least two adjacent horizontalshelves (e.g., shelves 622A and 622B) forming a substantially continuoussurface for product placement. The store shelves may include standardstore shelves or customized store shelves. A length of each store shelf622 may be at least 50 cm, less than 200 cm, or between 75 cm to 175 cm.In one embodiment, first housing 502 may be fixedly mounted on theretail shelving unit in a slit between two adjacent horizontal shelves.For example, first housing 502G may be fixedly mounted on retailshelving unit 620 in a slit between horizontal shelf 622B and horizontalshelf 622C. In another embodiment, first housing 502 may be fixedlymounted on a first shelf and second housing 504 may be fixedly mountedon a second shelf. For example, first housing 502I may be mounted onhorizontal shelf 622D and second housing 504I may be mounted onhorizontal shelf 622E. In another embodiment, first housing 502 may befixedly mounted on a retail shelving unit on a first side of ahorizontal shelf facing the opposing retail shelving unit and secondhousing 504 may be fixedly mounted on retail shelving unit 620 on asecond side of the horizontal shelf orthogonal to the first side. Forexample, first housing 502H may mounted on a first side 624 ofhorizontal shelf 622C next to a label and second housing 504H may bemounted on a second side 626 of horizontal shelf 622C that faces down(e.g., towards the ground or towards a lower shelf). In anotherembodiment, second housing 504 may be mounted closer to the back of thehorizontal shelf than to the front of the horizontal shelf. For example,second housing 504H may be fixedly mounted on horizontal shelf 622C onsecond side 626 closer to third side 628 of the horizontal shelf 622Cthan to first side 624. Third side 628 may be parallel to first side624. As mentioned above, data conduit 508 (e.g., data conduit 508H) mayhave an adjustable or selectable length for extending between firsthousing 502 and second housing 504. In one embodiment, when firsthousing 502H is fixedly mounted on first side 624, the length of dataconduit 508H may enable second housing 604H to be fixedly mounted onsecond side 626 closer to third side 628 than to first side 624.

As mentioned above, at least one processor contained in a single secondhousing 504 may control a plurality of image capture devices 506contained in a plurality of first housings 502 (e.g., system 500J). Insome embodiments, the plurality of image capture devices 506 may beconfigured for location on a single horizontal shelf and may be directedto substantially the same area of the opposing first retail shelvingunit (e.g., system 500D in FIG. 6A). In these embodiments, the imagedata acquired by the first image capture device and the second imagecapture device may enable a calculation of depth information (e.g.,based on image parallax information) associated with at least oneproduct positioned on an opposing retail shelving unit. For example,system 500J may have single second housing 504J with at least oneprocessor that may control a first image capturing device contained infirst housing 502J1 and a second image capturing device contained infirst housing 502J2. The distance d3 between the first image capturedevice contained in first housing 502J1 and the second image capturedevice contained in first housing 502J2 may be selected based on thedistance between retail shelving unit 620 and the opposing retailshelving unit (e.g., similar to d1 and/or d2). For example, distance d3may be at least 5 cm, at least 10 cm, at least 15 cm, less than 40 cm,less than 30 cm, between about 5 cm to about 20 cm, or between about 10cm to about 15 cm. In another example, d3 may be a function of d1 and/ord2, a linear function of d1 and/or d2, a function of d1*log(d1) and/ord2*log(d2) such as a1*d1*log(d1) for some constant a1, and so forth. Thedata from the first image capturing device contained in first housing502J1 and the second image capturing device contained in first housing502J2 may be used to estimate the number of products on a store shelf ofretail shelving unit 602. In related embodiments, system 500 may controla projector (e.g., projector 632) and image capture device 506 that areconfigured for location on a single store shelf or on two separate storeshelves. For example, projector 632 may be mounted on horizontal shelf622E and image capture device 5061 may be mounted on horizontal shelf622D. The image data acquired by image capture device 506 (e.g.,included in first housing 502I) may include reflections of lightpatterns projected from projector 632 on the at least one product and/orthe opposing retail shelving unit and may enable a calculation of depthinformation associated with at least one product positioned on theopposing retail shelving unit. The distance between projector 632 andthe image capture device contained in first housing 502I may be selectedbased on the distance between retail shelving unit 620 and the opposingretail shelving unit (e.g., similar to d1 and/or d2). For example, thedistance between the projector and the image capture device may be atleast 5 cm, at least 10 cm, at least 15 cm, less than 40 cm, less than30 cm, between about 5 cm to about 20 cm, or between about 10 cm toabout 15 cm. In another example, the distance between the projector andthe image capture device may be a function of d1 and/or d2, a linearfunction of d1 and/or d2, a function of d1*log(d1) and/or d2*log(d2)such as a1*d1*log(d1) for some constant a1, and so forth.

Consistent with the present disclosure, a central communication device630 may be located in retail store 105 and may be configured tocommunicate with server 135 (e.g., via an Internet connection). Thecentral communication device may also communicate with a plurality ofsystems 500 (for example, less than ten, ten, eleven, twelve, more thantwelve, and so forth). In some cases, at least one system of theplurality of systems 500 may be located in proximity to centralcommunication device 630. In the illustrated example, system 500F may belocated in proximity to central communication device 630. In someembodiments, at least some of systems 500 may communicate directly withat least one other system 500. The communications between some of theplurality of systems 500 may happen via a wired connection, such as thecommunications between system 500J and system 500I and thecommunications between system 500H and system 500G. Additionally oralternatively, the communications between some of the plurality ofsystems 500 may occur via a wireless connection, such as thecommunications between system 500G and system 500F and thecommunications between system 500I and system 500F. In some examples, atleast one system 500 may be configured to transmit captured image data(or information derived from the captured image data) to centralcommunication device 630 via at least two mediating systems 500, atleast three mediating systems 500, at least four mediating systems 500,or more. For example, system 500J may convey captured image data tocentral communication device 630 via system 500I and system 500F.

Consistent with the present disclosure, two (or more) systems 500 mayshare information to improve image acquisition. For example, system 500Jmay be configured to receive from a neighboring system 500I informationassociated with an event that system 500I had identified, and controlimage capture device 506 based on the received information. For example,system 500J may forgo image acquisition based on an indication fromsystem 500I that an object has entered or is about to enter its field ofview. Systems 500I and 500J may have overlapping fields of view ornon-overlapping fields of view. In addition, system 500J may alsoreceive (from system 500I) information that originates from centralcommunication device 630 and control image capture device 506 based onthe received information. For example, system 500I may receiveinstructions from central communication device 630 to capture an imagewhen suppler 115 inquiries about a specific product that is placed in aretail unit opposing system 500I. In some embodiments, a plurality ofsystems 500 may communicate with central communication device 630. Inorder to reduce or avoid network congestion, each system 500 mayidentify an available transmission time slot. Thereafter, each system500 may determine a default time slot for future transmissions based onthe identified transmission time slot.

FIG. 6C provides a diagrammatic representation of a retail shelving unit640 being captured by multiple systems 500 (e.g., system 500K and system500L) deployed on an opposing retail shelving unit (not shown). FIG. 6Cillustrates embodiments associated with the process of installingsystems 500 in retail store 105. To facilitate the installation ofsystem 500, each first housing 502 (e.g., first housing 502K) mayinclude an adjustment mechanism 642 for setting a field of view 644 ofimage capture device 506K such that the field of view 644 will at leastpartially encompass products placed both on a bottom shelf of retailshelving unit 640 and on a top shelf of retail shelving unit 640. Forexample, adjustment mechanism 642 may enable setting the position ofimage capture device 506K relative to first housing 502K. Adjustmentmechanism 642 may have at least two degrees of freedom to separatelyadjust manually (or automatically) the vertical field of view and thehorizontal field of view of image capture device 506K. In oneembodiment, the angle of image capture device 506K may be measured usingposition sensors associated with adjustment mechanism 642, and themeasured orientation may be used to determine if image capture device506K is positioned in the right direction. In one example, the output ofthe position sensors may be displayed on a handheld device of a storeassociate installing image capturing device 506K. Such an arrangementmay provide the store associate/installer with real time visual feedbackrepresentative of the field of view of an image acquisition device beinginstalled.

In addition to adjustment mechanism 642, first housing 502 may include afirst physical adapter (not shown) configured to operate with multipletypes of image capture device 506 and a second physical adapter (notshown) configured to operate with multiple types of lenses. Duringinstallation, the first physical adapter may be used to connect asuitable image capture device 506 to system 500 according to the levelof recognition requested (e.g., detecting a barcode from products,detecting text and price from labels, detecting different categories ofproducts, etc.). Similarly, during installation, the second physicaladapter may be used to associate a suitable lens to image capture device506 according to the physical conditions at the store (e.g., thedistance between the aisles, the horizontal field of view required fromimage capture device 506, and/or the vertical field of view requiredfrom image capture device 506). The second physical adapter provides thestore associate/installer the ability to select the focal length of lens312 during installation according to the distance between retailshelving units on opposite sides of an aisle (e.g., distance d1 and/ordistance d2 shown in FIG. 6A). In some embodiments, adjustment mechanism642 may include a locking mechanism to reduce the likelihood ofunintentional changes in the field of view of image capture device 506.Additionally or alternatively, the at least one processor contained insecond housing 504 may detect changes in the field of view of imagecapture device 506 and issue a warning when a change is detected, when achange larger than a selected threshold is detected, when a change isdetected for a duration longer than a selected threshold, and so forth.

In addition to adjustment mechanism 642 and the different physicaladapters, system 500 may modify the image data acquired by image capturedevice 506 based on at least one attribute associated with opposingretail shelving unit 640. Consistent with the present disclosure, the atleast one attribute associated with retail shelving unit 640 may includea lighting condition, the dimensions of opposing retail shelving unit640, the size of products displayed on opposing retail shelving unit640, the type of labels used on opposing retail shelving unit 640, andmore. In some embodiments, the attribute may be determined, based onanalysis of one or more acquired images, by at least one processorcontained in second housing 504. Alternatively, the attribute may beautomatically sensed and conveyed to the at least one processorcontained in second housing 504. In one example, the at least oneprocessor may change the brightness of captured images based on thedetected light conditions. In another example, the at least oneprocessor may modify the image data by cropping the image such that itwill include only the products on retail shelving unit (e.g., not toinclude the floor or the ceiling), only area of the shelving unitrelevant to a selected task (such as planogram compliance check), and soforth.

Consistent with the present disclosure, during installation, system 500may enable real-time display 646 of field of view 644 on a handhelddevice 648 of a user 650 installing image capturing device 506K. In oneembodiment, real-time display 646 of field of view 644 may includeaugmented markings 652 indicating a location of a field of view 654 ofan adjacent image capture device 506L. In another embodiment, real-timedisplay 646 of field of view 644 may include augmented markings 656indicating a region of interest in opposing retail shelving unit 640.The region of interest may be determined based on a planogram,identified product type, and/or part of retail shelving unit 640. Forexample, the region of interest may include products with a greaterlikelihood of planogram incompliance. In addition, system 500K mayanalyze acquired images to determine if field of view 644 includes thearea that image capturing device 506K is supposed to monitor (forexample, from labels on opposing retail shelving unit 640, products onopposing retail shelving unit 640, images captured from other imagecapturing devices that may capture other parts of opposing retailshelving unit 640 or capture the same part of opposing retail shelvingunit 640 but in a lower resolution or at a lower frequency, and soforth). In additional embodiments, system 500 may further comprise anindoor location sensor which may help determine if the system 500 ispositioned at the right location in retail store 105.

In some embodiments, an anti-theft device may be located in at least oneof first housing 502 and second housing 504. For example, the anti-theftdevice may include a specific RF label or a pin-tag radio-frequencyidentification device, which may be the same or similar to a type ofanti-theft device that is used by retail store 105 in which system 500is located. The RF label or the pin-tag may be incorporated within thebody of first housing 502 and second housing 504 and may not be visible.In another example, the anti-theft device may include a motion sensorwhose output may be used to trigger an alarm in the case of motion ordisturbance, in case of motion that is above a selected threshold, andso forth.

FIG. 7A includes a flowchart representing an exemplary method 700 foracquiring images of products in retail store 105 in accordance withexample embodiments of the present disclosure. For purposes ofillustration, in the following description, reference is made to certaincomponents of system 500 as deployed in the configuration depicted inFIG. 6A. It will be appreciated, however, that other implementations arepossible and that other configurations may be utilized to implement theexemplary method. It will also be readily appreciated that theillustrated method can be altered to modify the order of steps, deletesteps, or further include additional steps.

At step 702, the method includes fixedly mounting on first retailshelving unit 602 at least one first housing 502 containing at least oneimage capture device 506 such that an optical axis (e.g., optical axis606) of at least one image capture device 506 is directed to secondretail shelving unit 604. In one embodiment, fixedly mounting firsthousing 502 on first retail shelving unit 602 may include placing firsthousing 502 on a side of store shelf 622 facing second retail shelvingunit 604. In another embodiment, fixedly mounting first housing 502 onretail shelving unit 602 may include placing first housing 502 in a slitbetween two adjacent horizontal shelves. In some embodiments, the methodmay further include fixedly mounting on first retail shelving unit 602at least one projector (such as projector 632) such that light patternsprojected by the at least one projector are directed to second retailshelving unit 604. In one embodiment, the method may include mountingthe at least one projector to first retail shelving unit 602 at aselected distance to first housing 502 with image capture device 506. Inone embodiment, the selected distance may be at least 5 cm, at least 10cm, at least 15 cm, less than 40 cm, less than 30 cm, between about 5 cmto about 20 cm, or between about 10 cm to about 15 cm. In oneembodiment, the selected distance may be calculated according to adistance between to first retail shelving unit 602 and second retailshelving unit 604, such as d1 and/or d2, for example selecting thedistance to be a function of d1 and/or d2, a linear function of d1and/or d2, a function of d1*log(d1) and/or d2*log(d2) such asa1*d1*log(d1) for some constant a1, and so forth.

At step 704, the method includes fixedly mounting on first retailshelving unit 602 second housing 504 at a location spaced apart from theat least one first housing 502, second housing 504 may include at leastone processor (e.g., processing device 302). In one embodiment, fixedlymounting second housing 504 on the retail shelving unit may includeplacing second housing 504 on a different side of store shelf 622 thanthe side first housing 502 is mounted on.

At step 706, the method includes extending at least one data conduit 508between at least one first housing 502 and second housing 504. In oneembodiment, extending at least one data conduit 508 between at least onefirst housing 502 and second housing 504 may include adjusting thelength of data conduit 508 to enable first housing 502 to be mountedseparately from second housing 504. At step 708, the method includescapturing images of second retail shelving unit 604 using at least oneimage capture device 506 contained in at least one first housing 502(e.g., first housing 502A, first housing 502B, or first housing 502C).In one embodiment, the method further includes periodically capturingimages of products located on second retail shelving unit 604. Inanother embodiment the method includes capturing images of second retailshelving unit 604 after receiving a trigger from at least one additionalsensor in communication with system 500 (wireless or wired).

At step 710, the method includes transmitting at least some of thecaptured images from second housing 504 to a remote server (e.g., server135) configured to determine planogram compliance relative to secondretail shelving unit 604. In some embodiments, determining planogramcompliance relative to second retail shelving unit 604 may includedetermining at least one characteristic of planogram compliance based ondetected differences between the at least one planogram and the actualplacement of the plurality of product types on second retail shelvingunit 604. Consistent with the present disclosure, the characteristic ofplanogram compliance may include at least one of: product facing,product placement, planogram compatibility, price correlation, promotionexecution, product homogeneity, restocking rate, and planogramcompliance of adjacent products.

FIG. 7B provides a flowchart representing an exemplary method 720 foracquiring images of products in retail store 105, in accordance withexample embodiments of the present disclosure. For purposes ofillustration, in the following description, reference is made to certaincomponents of system 500 as deployed in the configuration depicted inFIG. 6A. It will be appreciated, however, that other implementations arepossible and that other configurations may be utilized to implement theexemplary method. It will also be readily appreciated that theillustrated method can be altered to modify the order of steps, deletesteps, or further include additional steps.

At step 722, at least one processor contained in a second housing mayreceive from at least one image capture device contained in at least onefirst housing fixedly mounted on a retail shelving unit a plurality ofimages of an opposing retail shelving unit. For example, at least oneprocessor contained in second housing 504A may receive from at least oneimage capture device 506 contained in first housing 502A (fixedlymounted on first retail shelving unit 602) a plurality of images ofsecond retail shelving unit 604. The plurality of images may be capturedand collected during a period of time (e.g., a minute, an hour, sixhours, a day, a week, or more).

At step 724, the at least one processor contained in the second housingmay analyze the plurality of images acquired by the at least one imagecapture device. In one embodiment, at least one processor contained insecond housing 504A may use any suitable image analysis technique (forexample, object recognition, object detection, image segmentation,feature extraction, optical character recognition (OCR), object-basedimage analysis, shape region techniques, edge detection techniques,pixel-based detection, artificial neural networks, convolutional neuralnetworks, etc.) to identify objects in the plurality of images. In oneexample, the at least one processor contained in second housing 504A maydetermine the number of products located in second retail shelving unit604. In another example, the at least one processor contained in secondhousing 504A may detect one or more objects in an area between firstretail shelving unit 602 and second retail shelving unit 604.

At step 726, the at least one processor contained in the second housingmay identify in the plurality of images a first image that includes arepresentation of at least a portion of an object located in an areabetween the retail shelving unit and the opposing retail shelving unit.In step 728, the at least one processor contained in the second housingmay identify in the plurality of images a second image that does notinclude any object located in an area between the retail shelving unitand the opposing retail shelving unit. In one example, the object in thefirst image may be an individual, such as a customer or a storeassociate. In another example, the object in the first image may be aninanimate object, such as carts, boxes, products, etc.

At step 730, the at least one processor contained in the second housingmay instruct a network interface contained in the second housing,fixedly mounted on the retail shelving unit separate from the at leastone first housing, to transmit the second image to a remote server andto avoid transmission of the first image to the remote server. Inaddition, the at least one processor may issue a notification when anobject blocks the field of view of the image capturing device for morethan a predefined period of time (e.g., at least 30 minutes, at least 75minutes, at least 150 minutes).

Embodiments of the present disclosure may automatically assesscompliance of one or more store shelves with a planogram. For example,embodiments of the present disclosure may use signals from one or moresensors to determine placement of one or more products on store shelves.The disclosed embodiments may also use one or more sensors to determineempty spaces on the store shelves. The placements and empty spaces maybe automatically assessed against a digitally encoded planogram. Aplanogram refers to any data structure or specification that defines atleast one product characteristic relative to a display structureassociated with a retail environment (such as store shelf or area of oneor more shelves). Such product characteristics may include, among otherthings, quantities of products with respect to areas of the shelves,product configurations or product shapes with respect to areas of theshelves, product arrangements with respect to areas of the shelves,product density with respect to areas of the shelves, productcombinations with respect to areas of the shelves, etc. Althoughdescribed with reference to store shelves, embodiments of the presentdisclosure may also be applied to end caps or other displays; bins,shelves, or other organizers associated with a refrigerator or freezerunits; or any other display structure associated with a retailenvironment.

The embodiments disclosed herein may use any sensors configured todetect one or more parameters associated with products (or a lackthereof). For example, embodiments may use one or more of pressuresensors, weight sensors, light sensors, resistive sensors, capacitivesensors, inductive sensors, vacuum pressure sensors, high pressuresensors, conductive pressure sensors, infrared sensors, photo-resistorsensors, photo-transistor sensors, photo-diodes sensors, ultrasonicsensors, or the like. Some embodiments may use a plurality of differentkinds of sensors, for example, associated with the same or overlappingareas of the shelves and/or associated with different areas of theshelves. Some embodiments may use a plurality of sensors configured tobe placed adjacent a store shelf, configured for location on the storeshelf, configured to be attached to, or configured to be integrated withthe store shelf. In some cases, at least part of the plurality ofsensors may be configured to be placed next to a surface of a storeshelf configured to hold products. For example, the at least part of theplurality of sensors may be configured to be placed relative to a partof a store shelf such that the at least part of the plurality of sensorsmay be positioned between the part of a store shelf and products placedon the part of the shelf. In another embodiment, the at least part ofthe plurality of sensors may be configured to be placed above and/orwithin and/or under the part of the shelf.

In one example, the plurality of sensors may include light detectorsconfigured to be located such that a product placed on the part of theshelf may block at least some of the ambient light from reaching thelight detectors. The data received from the light detectors may beanalyzed to detect a product or to identify a product based on the shapeof a product placed on the part of the shelf. In one example, the systemmay identify the product placed above the light detectors based on datareceived from the light detectors that may be indicative of at leastpart of the ambient light being blocked from reaching the lightdetectors. Further, the data received from the light detectors may beanalyzed to detect vacant spaces on the store shelf. For example, thesystem may detect vacant spaces on the store shelf based on the receiveddata that may be indicative of no product being placed on a part of theshelf. In another example, the plurality of sensors may include pressuresensors configured to be located such that a product placed on the partof the shelf may apply detectable pressure on the pressure sensors.Further, the data received from the pressure sensors may be analyzed todetect a product or to identify a product based on the shape of aproduct placed on the part of the shelf. In one example, the system mayidentify the product placed above the pressure sensors based on datareceived from the pressure sensors being indicative of pressure beingapplied on the pressure sensors. In addition, the data from the pressuresensors may be analyzed to detect vacant spaces on the store shelf, forexample based on the readings being indicative of no product beingplaced on a part of the shelf, for example, when the pressure readingsare below a selected threshold. Consistent with the present disclosure,inputs from different types of sensors (such as pressure sensors, lightdetectors, etc.) may be combined and analyzed together, for example todetect products placed on a store shelf, to identify shapes of productsplaced on a store shelf, to identify types of products placed on a storeshelf, to identify vacant spaces on a store shelf, and so forth.

With reference to FIG. 8A and consistent with the present disclosure, astore shelf 800 may include a plurality of detection elements, e.g.,detection elements 801A and 801B. In the example of FIG. 8A, detectionelements 801A and 801B may comprise pressure sensors and/or other typeof sensors for measuring one or more parameters (such as resistance,capacitance, or the like) based on physical contact (or lack thereof)with products, e.g., product 803A and product 803B. Additionally oralternatively, detection elements configured to measure one or moreparameters (such as current induction, magnetic induction, visual orother electromagnetic reflectance, visual or other electromagneticemittance, or the like) may be included to detect products based onphysical proximity (or lack thereof) to products. Consistent with thepresent disclosure, the plurality of detection elements may beconfigured for location on shelf 800. The plurality of detectionelements may be configured to detect placement of products when theproducts are placed above at least part of the plurality of detectionelements. Some embodiments of the disclosure, however, may be performedwhen at least some of the detection elements may be located next toshelf 800 (e.g., for magnetometers or the like), across from shelf 800(e.g., for image sensors or other light sensors, light detection andranging (LIDAR) sensors, radio detection and ranging (RADAR) sensors, orthe like), above shelf 800 (e.g., for acoustic sensors or the like),below shelf 800 (e.g., for pressure sensors or the like), or any otherappropriate spatial arrangement. Although depicted as standalone unitsin the example of FIG. 8A, the plurality of detection elements may formpart of a fabric (e.g., a smart fabric or the like), and the fabric maybe positioned on a shelf to take measurements. For example, two or moredetection elements may be integrated together into a single structure(e.g., disposed within a common housing, integrated together within afabric or mat, etc.). In some examples, detection elements (such asdetection elements 801A and 801B) may be placed adjacent to (or placedon) store shelves as described above. Some examples of detectionelements may include pressure sensors and/or light detectors configuredto be placed above and/or within and/or under a store shelf as describedabove.

Detection elements associated with shelf 800 may be associated withdifferent areas of shelf 800. For example, detection elements 801A and801B are associated with area 805A while other detection elements areassociated with area 805B. Although depicted as rows, areas 805A and805B may comprise any areas of shelf 800, whether contiguous (e.g., asquare, a rectangular, or other regular or irregular shape) or not(e.g., a plurality of rectangles or other regular and/or irregularshapes). Such areas may also include horizontal regions between shelves(as shown in FIG. 8A) or may include vertical regions that include areaof multiple different shelves (e.g., columnar regions spanning overseveral different horizontally arranged shelves). In some examples, theareas may be part of a single plane. In some examples, each area may bepart of a different plane. In some examples, a single area may be partof a single plane or be divided across multiple planes.

One or more processors (e.g., processing device 202) configured tocommunicate with the detection elements (e.g., detection elements 801Aand 801B) may detect first signals associated with a first area (e.g.,areas 805A and/or 805B) and second signals associated with a secondarea. In some embodiments, the first area may, in part, overlap with thesecond area. For example, one or more detection elements may beassociated with the first area as well as the second area and/or one ormore detection elements of a first type may be associated with the firstarea while one or more detection elements of a second type may beassociated with the second area overlapping, at least in part, the firstarea. In other embodiments, the first area and the second area may bespatially separate from each other.

The one or more processors may, using the first and second signals,determine that one or more products have been placed in the first areawhile the second area includes at least one empty area. For example, ifthe detection elements include pressure sensors, the first signals mayinclude weight signals that match profiles of particular products (suchas the mugs or plates depicted in the example of FIG. 8A), and thesecond signals may include weight signals indicative of the absence ofproducts (e.g., by being equal to or within a threshold of a defaultvalue such as atmospheric pressure or the like). The disclosed weightsignals may be representative of actual weight values associated with aparticular product type or, alternatively, may be associated with arelative weight value sufficient to identify the product and/or toidentify the presence of a product. In some cases, the weight signal maybe suitable for verifying the presence of a product regardless ofwhether the signal is also sufficient for product identification. Inanother example, if the detection elements include light detectors (asdescribed above), the first signals may include light signals that matchprofiles of particular products (such as the mugs or plates depicted inthe example of FIG. 8A), and the second signals may include lightsignals indicative of the absence of products (e.g., by being equal toor within a threshold of a default value such as values corresponding toambient light or the like). For example, the first light signals may beindicative of ambient light being blocked by particular products, whilethe second light signals may be indicative of no product blocking theambient light. The disclosed light signals may be representative ofactual light patterns associated with a particular product type or,alternatively, may be associated with light patterns sufficient toidentify the product and/or to identify the presence of a product.

The one or more processors may similarly process signals from othertypes of sensors. For example, if the detection elements includeresistive or inductive sensors, the first signals may includeresistances, voltages, and/or currents that match profiles of particularproducts (such as the mugs or plates depicted in the example of FIG. 8Aor elements associated with the products, such as tags, etc.), and thesecond signals may include resistances, voltages, and/or currentsindicative of the absence of products (e.g., by being equal to or withina threshold of a default value such as atmospheric resistance, a defaultvoltage, a default current, corresponding to ambient light, or thelike). In another example, if the detection elements include acoustics,LIDAR, RADAR, or other reflective sensors, the first signals may includepatterns of returning waves (whether sound, visible light, infraredlight, radio, or the like) that match profiles of particular products(such as the mugs or plates depicted in the example of FIG. 8A), and thesecond signals may include patterns of returning waves (whether sound,visible light, infrared light, radio, or the like) indicative of theabsence of products (e.g., by being equal to or within a threshold of apattern associated with an empty shelf or the like).

Any of the profile matching described above may include direct matchingof a subject to a threshold. For example, direct matching may includetesting one or more measured values against the profile value(s) withina margin of error; mapping a received pattern onto a profile patternwith a residual having a maximum, minimum, integral, or the like withinthe margin of error; performing an autocorrelation, Fourier transform,convolution, or other operation on received measurements or a receivedpattern and comparing the resultant values or function against theprofile within a margin of error; or the like. Additionally oralternatively, profile matching may include fuzzy matching betweenmeasured values and/or patterns and a database of profiles such that aprofile with a highest level of confidence according to the fuzzysearch. Moreover, as depicted in the example of FIG. 8A, products, suchas product 803B, may be stacked and thus associated with a differentprofile when stacked than when standalone.

Any of the profile matching described above may include use of one ormore machine learning techniques. For example, one or more artificialneural networks, random forest models, or other models trained onmeasurements annotated with product identifiers may process themeasurements from the detection elements and identify productstherefrom. In such embodiments, the one or more models may useadditional or alternative input, such as images of the shelf (e.g., fromcapturing devices 125 of FIGS. 4A-4C explained above) or the like.

Based on detected products and/or empty spaces, determined using thefirst signals and second signals, the one or more processors maydetermine one or more aspects of planogram compliance. For example, theone or more processors may identify products and their locations on theshelves, determine quantities of products within particular areas (e.g.,identifying stacked or clustered products), identify facing directionsassociated with the products (e.g., whether a product is outward facing,inward facing, askew, or the like), or the like. Identification of theproducts may include identifying a product type (e.g., a bottle of soda,a loaf of broad, a notepad, or the like) and/or a product brand (e.g., aCoca-Cola® bottle instead of a Sprite® bottle, a Starbucks® coffeetumbler instead of a Tervis® coffee tumbler, or the like). Productfacing direction and/or orientation, for example, may be determinedbased on a detected orientation of an asymmetric shape of a product baseusing pressure sensitive pads, detected density of products, etc. Forexample, the product facing may be determined based on locations ofdetected product bases relative to certain areas of a shelf (e.g., alonga front edge of a shelf), etc. Product facing may also be determinedusing image sensors, light sensors, or any other sensor suitable fordetecting product orientation.

The one or more processors may generate one or more indicators of theone or more aspects of planogram compliance. For example, an indicatormay comprise a data packet, a data file, or any other data structureindicating any variations from a planogram, e.g., with respect toproduct placement such as encoding intended coordinates of a product andactual coordinates on the shelf, with respect to product facingdirection and/or orientation such as encoding indicators of locationsthat have products not facing a correct direction and/or in an undesiredorientation, or the like.

In addition to or as an alternative to determining planogram compliance,the one or more processors may detect a change in measurements from oneor more detection elements. Such measurement changes may trigger aresponse. For example, a change of a first type may trigger capture ofat least one image of the shelf (e.g., using capturing devices 125 ofFIGS. 4A-4C explained above) while a detected change of a second typemay cause the at least one processor to forgo such capture. A first typeof change may, for example, indicate the moving of a product from onelocation on the shelf to another location such that planogram compliancemay be implicated. In such cases, it may be desired to capture an imageof the product rearrangement in order to assess or reassess productplanogram compliance. In another example, a first type of change mayindicate the removal of a product from the shelf, e.g., by a storeassociate due to damage, by a customer to purchase, or the like. On theother hand, a second type of change may, for example, indicate theremoval and replacement of a product to the same (within a margin oferror) location on the shelf, e.g., by a customer to inspect the item.In cases where products are removed from a shelf, but then replaced onthe shelf (e.g., within a particular time window), the system may forgoa new image capture, especially if the replaced product is detected in alocation similar to or the same as its recent, original position.

With reference to FIG. 8B and consistent with the present disclosure, astore shelf 850 may include a plurality of detection elements, e.g.,detection elements 851A and 851B. In the example of FIG. 8B, detectionelements 851A and 851B may comprise light sensors and/or other sensorsmeasuring one or more parameters (such as visual or otherelectromagnetic reflectance, visual or other electromagnetic emittance,or the like) based on electromagnetic waves from products, e.g., product853A and product 853B. Additionally or alternatively, as explained abovewith respect to FIG. 8B, detection elements 851A and 851B may comprisepressure sensors, other sensors measuring one or more parameters (suchas resistance, capacitance, or the like) based on physical contact (orlack thereof) with the products, and/or other sensors that measure oneor more parameters (such as current induction, magnetic induction,visual or other electromagnetic reflectance, visual or otherelectromagnetic emittance, or the like) based on physical proximity (orlack thereof) to products.

Moreover, although depicted as located on shelf 850, some detectionelements may be located next to shelf 850 (e.g., for magnetometers orthe like), across from shelf 850 (e.g., for image sensors or other lightsensors, light detection and ranging (LIDAR) sensors, radio detectionand ranging (RADAR) sensors, or the like), above shelf 850 (e.g., foracoustic sensors or the like), below shelf 850 (e.g., for pressuresensors, light detectors, or the like), or any other appropriate spatialarrangement. Further, although depicted as standalone in the example ofFIG. 8B, the plurality of detection elements may form part of a fabric(e.g., a smart fabric or the like), and the fabric may be positioned ona shelf to take measurements.

Detection elements associated with shelf 850 may be associated withdifferent areas of shelf 850, e.g., area 855A, area 855B, or the like.Although depicted as rows, areas 855A and 855B may comprise any areas ofshelf 850, whether contiguous (e.g., a square, a rectangular, or otherregular or irregular shape) or not (e.g., a plurality of rectangles orother regular and/or irregular shapes).

One or more processors (e.g., processing device 202) in communicationwith the detection elements (e.g., detection elements 851A and 851B) maydetect first signals associated with a first area and second signalsassociated with a second area. Any of the processing of the first andsecond signals described above with respect to FIG. 8A may similarly beperformed for the configuration of FIG. 8B.

In both FIGS. 8A and 8B, the detection elements may be integral to theshelf, part of a fabric or other surface configured for positioning onthe shelf, or the like. Power and/or data cables may form part of theshelf, the fabric, the surface, or be otherwise connected to thedetection elements. Additionally or alternatively, as depicted in FIGS.8A and 8B, individual sensors may be positioned on the shelf. Forexample, the power and/or data cables may be positioned under the shelfand connected through the shelf to the detection elements. In anotherexample, power and/or data may be transmitted wirelessly to thedetection elements (e.g., to wireless network interface controllersforming part of the detection elements). In yet another example, thedetection elements may include internal power sources (such as batteriesor fuel cells).

With reference to FIG. 9 and consistent with the present disclosure, thedetection elements described above with reference to FIGS. 8A and 8B maybe arranged on rows of the shelf in any appropriate configuration. Allof the arrangements of FIG. 9 are shown as a top-down view of a row(e.g., area 805A, area 805B, area 855A, area 855B, or the like) on theshelf. For example, arrangements 910 and 940 are both uniformdistributions of detection elements within a row. However, arrangement910 is also uniform throughout the depth of the row while arrangement940 is staggered. Both arrangements may provide signals that representproducts on the shelf in accordance with spatially uniform measurementlocations. As further shown in FIG. 9, arrangements 920, 930, 950, and960 cluster detection elements near the front (e.g., a facing portion)of the row. Arrangement 920 includes detection elements at a frontportion while arrangement 930 includes defection elements in a largerportion of the front of the shelf. Such arrangements may save power andprocessing cycles by having fewer detection elements on a back portionof the shelf. Arrangements 950 and 960 include some detection elementsin a back portion of the shelf but these elements are arranged lessdense than detection elements in the front. Such arrangements may allowfor detections in the back of the shelf (e.g., a need to restockproducts, a disruption to products in the back by a customer or storeassociate, or the like) while still using less power and fewerprocessing cycles than arrangements 910 and 940. And, such arrangementsmay include a higher density of detection elements in regions of theshelf (e.g., a front edge of the shelf) where product turnover rates maybe higher than in other regions (e.g., at areas deeper into a shelf),and/or in regions of the shelf where planogram compliance is especiallyimportant.

FIG. 10A is a flow chart, illustrating an exemplary method 1000 formonitoring planogram compliance on a store shelf, in accordance with thepresently disclosed subject matter. It is contemplated that method 1000may be used with any of the detection element arrays discussed abovewith reference to, for example, FIGS. 8A, 8B and 9. The order andarrangement of steps in method 1000 is provided for purposes ofillustration. As will be appreciated from this disclosure, modificationsmay be made to process 1000, for example, adding, combining, removing,and/or rearranging one or more steps of process 1000.

Method 1000 may include a step 1005 of receiving first signals from afirst subset of detection elements (e.g., detection elements 801A and801B of FIG. 8A) from among the plurality of detection elements afterone or more of a plurality of products (e.g., products 803A and 803B)are placed on at least one area of the store shelf associated with thefirst subset of detection elements. As explained above with respect toFIGS. 8A and 8B, the plurality of detection elements may be embeddedinto a fabric configured to be positioned on the store shelf.Additionally or alternatively, the plurality of detection elements maybe configured to be integrated with the store shelf. For example, anarray of pressure sensitive elements (or any other type of detector) maybe fabricated as part of the store shelf. In some examples, theplurality of detection elements may be configured to placed adjacent to(or located on) store shelves, as described above.

As described above with respect to arrangements 910 and 940 of FIG. 9,the plurality of detection elements may be substantially uniformlydistributed across the store shelf. Alternatively, as described abovewith respect to arrangements 920, 930, 950, and 960 of FIG. 9, theplurality of detection elements may be distributed relative to the storeshelf such that a first area of the store shelf has a higher density ofdetection elements than a second area of the store shelf. For example,the first area may comprise a front portion of the shelf, and the secondarea may comprise a back portion of the shelf.

In some embodiments, such as those including pressure sensors or othercontact sensors as depicted in the example of FIG. 8A, step 1005 mayinclude receiving the first signals from the first subset of detectionelements as the plurality of products are placed above the first subsetof detection elements. In some embodiments where the plurality ofdetection elements includes pressure detectors, the first signals may beindicative of pressure levels detected by pressure detectorscorresponding to the first subset of detection elements after one ormore of the plurality of products are placed on the at least one area ofthe store shelf associated with the first subset of detection elements.For example, the first signals may be indicative of pressure levelsdetected by pressure detectors corresponding to the first subset ofdetection elements after stocking at least one additional product abovea product previously positioned on the shelf, removal of a product fromthe shelf, or the like. In other embodiments where the plurality ofdetection elements includes light detectors, the first signals may beindicative of light measurements made with respect to one or more of theplurality of products placed on the at least one area of the store shelfassociated with the first subset of detection elements. Specifically,the first signals may be indicative of at least part of the ambientlight being blocked from reaching the light detectors by the one or moreof the plurality of products.

In embodiments including proximity sensors as depicted in the example ofFIG. 8B, step 1005 may include receiving the first signals from thefirst subset of detection elements as the plurality of products areplaced below the first subset of detection elements. In embodimentswhere the plurality of detection elements include proximity detectors,the first signals may be indicative of proximity measurements made withrespect to one or more of the plurality of products placed on the atleast one area of the store shelf associated with the first subset ofdetection elements.

Method 1000 may include step 1010 of using the first signals to identifyat least one pattern associated with a product type of the plurality ofproducts. For example, any of the pattern matching techniques describedabove with respect to FIGS. 8A and 8B may be used for identification. Apattern associated with a product type may include a pattern (e.g., acontinuous ring, a discontinuous ring of a certain number of points, acertain shape, etc.) associated with a base of a single product. Thepattern associated with a product type may also be formed by a group ofproducts. For example, a six pack of soda cans may be associated with apattern including a 2×3 array of continuous rings associated with thesix cans of that product type. Additionally, a grouping of two literbottles may form a detectable pattern including an array (whetheruniform, irregular, or random) of discontinuous rings of pressurepoints, where the rings have a diameter associated with a particular2-liter product. Various other types of patterns may also be detected(e.g., patterns associated with different product types arrangedadjacent to one another, patterns associated with solid shapes (such asa rectangle of a boxed product), etc.). In another example, anartificial neural network configured to recognize product types may beused to analyze the signals received by step 1005 (such as signals frompressure sensors, from light detectors, from contact sensors, and soforth) to determine product types associated with products placed on anarea of a shelf (such as an area of a shelf associated with the firstsubset of detection elements). In yet another example, a machinelearning algorithm trained using training examples to recognize producttypes may be used to analyze the signals received by step 1005 (such assignals from pressure sensors, from light detectors, from contactsensors, and so forth) to determine product types associated withproducts placed on an area of a shelf (such as an area of a shelfassociated with the first subset of detection elements).

In some embodiments, step 1010 may further include accessing a memorystoring data (e.g., memory device 226 of FIG. 2 and/or memory device 314of FIG. 3A) associated with patterns of different types of products. Insuch embodiments, step 1010 may include using the first signals toidentify at least one product of a first type using a first pattern (ora first product model) and at least one product of a second type using asecond pattern (or a second product model). For example, the first typemay include one brand (such as Coca-Cola® or Folgers®) while the secondtype may include another brand (such as Pepsi® or Maxwell House®). Inthis example, a size, shape, point spacing, weight, resistance or otherproperty of the first brand may be different from that of the secondbrand such that the detection elements may differentiate the brands.Such characteristics may also be used to differentiate like-branded, butdifferent products from one another (e.g., a 12-ounce can of Coca Cola,versus a 16 oz bottle of Coca Cola, versus a 2-liter bottle of CocaCola). For example, a soda may have a base detectable by a pressuresensitive pad as a continuous ring. Further, the can of soda may beassociated with a first weight signal having a value recognizable asassociated with such a product. A 16 ounce bottle of soda may beassociated with a base having four or five pressure points, which apressure sensitive pad may detect as arranged in a pattern associatedwith a diameter typical of such a product. The 16 ounce bottle of sodamay also be associated with a second weight signal having a value higherthan the weight signal associated with the 12 ounce can of soda. Furtherstill, a 2 liter bottle of soda may be associated with a base having aring, four or five pressure points, etc. that a pressure sensitive padmay detect as arranged in a pattern associated with a diameter typicalof such a product. The 2 liter bottle of soda may be associated with aweight signal having a value higher than the weight signal associatedwith the 12 ounce can of soda and 16 ounce bottle of soda.

In the example of FIG. 8B, the different bottoms of product 853A andproduct 853B may be used to differentiate the products from each other.For example, detection elements such as pressure sensitive pads may beused to detect a product base shape and size (e.g., ring, pattern ofpoints, asymmetric shape, base dimensions, etc.). Such a base shape andsize may be used (optionally, together with one or more weight signals)to identify a particular product. The signals may also be used toidentify and/or distinguish product types from one another. For example,a first type may include one category of product (such as soda cans)while a second type may include a different category of product (such asnotepads). In another example, detection elements such as lightdetectors may be used to detect a product based on a pattern of lightreadings indicative of a product blocking at least part of the ambientlight from reaching the light detectors. Such pattern of light readingsmay be used to identify product type and/or product category and/orproduct shape. For example, products of a first type may block a firstsubset of light frequencies of the ambient light from reaching the lightdetectors, while products of a second type may block a second subset oflight frequencies of the ambient light from reaching the light detectors(the first subset and second subset may differ). In this case the typeof the products may be determined based on the light frequenciesreaching the light detectors. In another example, products of a firsttype may have a first shape of shades and therefore may block ambientlight from reaching light detectors arranged in one shape, whileproducts of a second type may have a second shape of shades andtherefore may block ambient light from reaching light detectors arrangedin another shape. In this case the type of the products may bedetermined based on the shape of blocked ambient light. Any of thepattern matching techniques described above may be used for theidentification.

Additionally or alternatively, step 1010 may include using the at leastone pattern to determine a number of products placed on the at least onearea of the store shelf associated with the first subset of detectionelements. For example, any of the pattern matching techniques describedabove may be used to identify the presence of one or more product typesand then to determine the number of products of each product type (e.g.,by detecting a number of similarly sized and shaped product bases andoptionally by detecting weight signals associated with each detectedbase). In another example, an artificial neural network configured todetermine the number of products of selected product types may be usedto analyze the signals received by step 1005 (such as signals frompressure sensors, from light detectors, from contact sensors, and soforth) to determine the number of products of selected product typesplaced on an area of a shelf (such as an area of a shelf associated withthe first subset of detection elements). In yet another example, amachine learning algorithm trained using training examples to determinethe number of products of selected product types may be used to analyzethe signals received by step 1005 (such as signals from pressuresensors, from light detectors, from contact sensors, and so forth) todetermine the number of products of selected product types placed on anarea of a shelf (such as an area of a shelf associated with the firstsubset of detection elements). Additionally or alternatively, step 1010may include extrapolating from a stored pattern associated with a singleproduct (or type of product) to determine the number of productsmatching the first signals. In such embodiments, step 1010 may furtherinclude determining, for example based on product dimension data storedin a memory, a number of additional products that may be placed on theat least one area of the store shelf associated with the second subsetof detection elements. For example, step 1010 may include extrapolatingbased on stored dimensions of each product and stored dimensions of theshelf area to determine an area and/or volume available for additionalproducts. Step 1010 may further include extrapolation of the number ofadditional products based on the stored dimensions of each product anddetermined available area and/or volume.

Method 1000 may include step 1015 of receiving second signals from asecond subset of detection elements (e.g., detection elements 851A and851B of FIG. 8B) from among the plurality of detection elements, thesecond signals being indicative of no products being placed on at leastone area of the store shelf associated with the second subset ofdetection elements. Using this information, method 1000 may include step1020 of using the second signals to determine at least one empty spaceon the store shelf. For example, any of the pattern matching techniquesdescribed above may be used to determine that the second signals includedefault values or other values indicative of a lack of product incertain areas associated with a retail store shelf. A default value maybe include, for example, a pressure signal associated with an un-loadedpressure sensor or pressure sensitive mat, indicating that no product islocated in a certain region of a shelf. In another example, a defaultvalue may include signals from light detectors corresponding to ambientlight, indicating that no product is located in a certain region of ashelf.

Method 1000 may include step 1025 of determining, based on the at leastone pattern associated with a detected product and the at least oneempty space, at least one aspect of planogram compliance. As explainedabove with respect to FIGS. 8A and 8B, the aspect of planogramcompliance may include the presence or absence of particular products(or brands), locations of products on the shelves, quantities ofproducts within particular areas (e.g., identifying stacked or clusteredproducts), facing directions associated with the products (e.g., whethera product is outward facing, inward facing, askew, or the like), or thelike. A planogram compliance determination may be made, for example, bydetermining a number of empty spaces on a shelf and determining alocation of the empty spaces on a shelf. The planogram determination mayalso include determining weight signal magnitudes associated withdetected products at the various detected non-empty locations. Thisinformation may be used by the one or more processors in determiningwhether a product facing specification has been satisfied (e.g., whethera front edge of a shelf has a suitable number of products or suitabledensity of products), whether a specified stacking density has beenachieved (e.g., by determining a pattern of detected products and weightsignals of the detected products to determine how many products arestacked at each location), whether a product density specification hasbeen achieved (e.g., by determining a ratio of empty locations toproduct-present locations), whether products of a selected product typeare located in a selected area of the shelf, whether all productslocated in a selected area of the shelf are of a selected product type,whether a selected number of products (or a selected number of productsof a selected product type) are located in a selected area of the shelf,whether products located in a selected area of a shelf are positioned ina selected orientation, or whether any other aspect of one or moreplanograms has been achieved.

For example, the at least one aspect may include product homogeneity,and step 1025 may further include counting occurrences where a productof the second type is placed on an area of the store shelf associatedwith the first type of product. For example, by accessing a memoryincluding base patterns (or any other type of pattern associated withproduct types, such as product models), the at least one processor maydetect different products and product types. A product of a first typemay be recognized based on a first pattern, and product of a second typemay be recognized based on a second, different pattern (optionally alsobased on weight signal information to aid in differentiating betweenproducts). Such information may be used, for example, to monitor whethera certain region of a shelf includes an appropriate or intended productor product type. Such information may also be useful in determiningwhether products or product types have been mixed (e.g., producthomogeneity). Regarding planogram compliance, detection of differentproducts and their relative locations on a shelf may aid in determiningwhether a product homogeneity value, ratio, etc. has been achieved. Forexample, the at least one processor may count occurrences where aproduct of a second type is placed on an area of the store shelfassociated with a product of a first type.

Additionally or alternatively, the at least one aspect of planogramcompliance may include a restocking rate, and step 1025 may furtherinclude determining the restocking rate based on a sensed rate at whichproducts are added to the at least one area of the store shelfassociated with the second subset of detection elements. Restocking ratemay be determined, for example, by monitoring a rate at which detectionelement signals change as products are added to a shelf (e.g., whenareas of a pressure sensitive pad change from a default value to aproduct-present value).

Additionally or alternatively, the at least one aspect of planogramcompliance may include product facing, and step 1025 may further includedetermining the product facing based on a number of products determinedto be placed on a selected area of the store shelf at a front of thestore shelf. Such product facing may be determined by determining anumber of products along a certain length of a front edge of a storeshelf and determining whether the number of products complies with, forexample, a specified density of products, a specified number ofproducts, and so forth.

Step 1025 may further include transmitting an indicator of the at leastone aspect of planogram compliance to a remote server. For example, asexplained above with respect to FIGS. 8A and 8B, the indicator maycomprise a data packet, a data file, or any other data structureindicating any variations from a planogram, e.g., with respect toproduct (or brand) placement, product facing direction, or the like. Theremote server may include one or more computers associated with a retailstore (e.g., so planogram compliance may be determined on a local basiswithin a particular store), one or more computers associated with aretail store evaluation body (e.g., so planogram compliance may bedetermined across a plurality of retail stores), one or more computersassociated with a product manufacturer, one or more computers associatedwith a supplier (such as supplier 115), one or more computers associatedwith a market research entity (such as market research entity 110), etc.

Method 1000 may further include additional steps. For example, method1000 may include identifying a change in at least one characteristicassociated with one or more of the first signals (e.g., signals from afirst group or type of detection elements), and in response to theidentified change, triggering an acquisition of at least one image ofthe store shelf. The acquisition may be implemented by activating one ormore of capturing devices 125 of FIGS. 4A-4C, as explained above. Forexample, the change in at least one characteristic associated with oneor more of the first signals may be indicative of removal of at leastone product from a location associated with the at least one area of thestore shelf associated with the first subset of detection elements.Accordingly, method 1000 may include triggering the acquisition todetermine whether restocking, reorganizing, or other intervention isrequired, e.g., to improve planogram compliance. Thus, method 1000 mayinclude identifying a change in at least one characteristic associatedwith one or more of the first signals; and in response to the identifiedchange, trigger a product-related task for a store associate of theretail store.

Additionally or alternatively, method 1000 may be combined with method1050 of FIG. 10B, described below, such that step 1055 is performed anytime after step 1005.

FIG. 10B is a flow chart, illustrating an exemplary method 1050 fortriggering image capture of a store shelf, in accordance with thepresently disclosed subject matter. It is contemplated that method 1050may be used in conjunction with any of the detection element arraysdiscussed above with reference to, for example, FIGS. 8A, 8B and 9. Theorder and arrangement of steps in method 1050 is provided for purposesof illustration. As will be appreciated from this disclosure,modifications may be made to process 1050, for example, adding,combining, removing, and/or rearranging one or more steps of process1050.

Method 1050 may include a step 1055 of determining a change in at leastone characteristic associated with one or more first signals. Forexample, the first signals may have been captured as part of method 1000of FIG. 10A, described above. For example, the first signals may includepressure readings when the plurality of detection elements includespressure sensors, contact information when the plurality of detectionelements includes contact sensors, light readings when the plurality ofdetection elements includes light detectors (for example, from lightdetectors configured to be placed adjacent to (or located on) a surfaceof a store shelf configured to hold products, as described above), andso forth.

Method 1050 may include step 1060 of using the first signals to identifyat least one pattern associated with a product type of the plurality ofproducts. For example, any of the pattern matching techniques describedabove with respect to FIGS. 8A, 8B, and step 1010 may be used foridentification.

Method 1050 may include step 1065 of determining a type of eventassociated with the change. For example, a type of event may include aproduct removal, a product placement, movement of a product, or thelike.

Method 1050 may include step 1070 of triggering an acquisition of atleast one image of the store shelf when the change is associated with afirst event type. For example, a first event type may include removal ofa product, moving of a product, or the like, such that the first eventtype may trigger a product-related task for a store associate of theretail store depending on analysis of the at least one image. Theacquisition may be implemented by activating one or more of capturingdevices 125 of FIGS. 4A-4C, as explained above. In some examples, thetriggered acquisition may include an activation of at least oneprojector (such as projector 632). In some examples, the triggeredacquisition may include acquisition of color images, depth images,stereo images, active stereo images, time of flight images, LIDARimages, RADAR images, and so forth.

Method 1050 may include a step (not shown) of forgoing the acquisitionof at least one image of the store shelf when the change is associatedwith a second event type. For example, a second event type may includereplacement of a removed product by a customer, stocking of a shelf by astore associate, or the like. As another example, a second event typemay include removal, placement, or movement of a product that isdetected within a margin of error of the detection elements and/ordetected within a threshold (e.g., removal of only one or two products;movement of a product by less than 5 cm, 20 cm, or the like; moving of afacing direction by less than 10 degrees; or the like), such that noimage acquisition is required.

FIGS. 11A-11E illustrate example outputs based on data automaticallyderived from machine processing and analysis of images captured inretail store 105 according to disclosed embodiments. FIG. 11Aillustrates an optional output for market research entity 110. FIG. 11Billustrates an optional output for supplier 115. FIGS. 11C and 11Dillustrate optional outputs for store associates of retail store 105.And FIG. 11E illustrates optional outputs for user 120.

FIG. 11A illustrates an example graphical user interface (GUI) 500 foroutput device 145A, representative of a GUI that may be used by marketresearch entity 110. Consistent with the present disclosure, marketresearch entity 110 may assist supplier 115 and other stakeholders inidentifying emerging trends, launching new products, and/or developingmerchandising and distribution plans across a large number of retailstores 105. By doing so, market research entity 110 may assist supplier115 in growing product presence and maximizing or increasing new productsales. As mentioned above, market research entity 110 may be separatedfrom or part of supplier 115. To successfully launch a new product,supplier 115 may use information about what really happens in retailstore 105. For example, supplier 115 may want to monitor how marketingplans are being executed and to learn what other competitors are doingrelative to certain products or product types. Embodiments of thepresent disclosure may allow market research entity 110 and suppliers115 to continuously monitor product-related activities at retail stores105 (e.g., using system 100 to generate various metrics or informationbased on automated analysis of actual, timely images acquired from theretail stores). For example, in some embodiments, market research entity110 may track how quickly or at what rate new products are introduced toretail store shelves, identify new products introduced by variousentities, assess a supplier's brand presence across different retailstores 105, among many other potential metrics.

In some embodiments, server 135 may provide market research entity 110with information including shelf organization, analysis of skewproductivity trends, and various reports aggregating information onproducts appearing across large numbers of retail stores 105. Forexample, as shown in FIG. 11A, GUI 1100 may include a first display area1102 for showing a percentage of promotion campaign compliance indifferent retail stores 105. GUI 1100 may also include a second displayarea 1104 showing a graph illustrating sales of a certain productrelative to the percentage of out of shelf. GUI 1100 may also include athird display area 1106 showing actual measurements of different factorsrelative to target goals (e.g., planogram compliance, restocking rate,price compliance, and other metrics). The provided information mayenable market research entity 110 to give supplier 115 informed shelvingrecommendations and fine-tune promotional strategies according toin-store marketing trends, to provide store managers with a comparisonof store performances in comparison to a group of retail stores 105 orindustry wide performances, and so forth.

FIG. 11B illustrates an example GUI 1110 for output device 145B used bysupplier 115. Consistent with the present disclosure, server 135 may usedata derived from images captured in a plurality of retail stores 105 torecommend a planogram, which often determines sales success of differentproducts. Using various analytics and planogram productivity measures,server 135 may help supplier 115 to determine an effective planogramwith assurances that most if not all retail stores 105 can execute theplan. For example, the determined planogram may increase the probabilitythat inventory is available for each retail store 105 and may bedesigned to decrease costs or to keep costs within a budget (such asinventory costs, restocking costs, shelf space costs, etc.). Server 135may also provide pricing recommendations based on the goals of supplier115 and other factors. In other words, server 135 may help supplier 115understand how much room to reserve for different products and how tomake them available for favorable sales and profit impact (for example,by choosing the size of the shelf dedicated to a selected product, thelocation of the shelf, the height of the shelf, the neighboringproducts, and so forth). In addition, server 135 may monitor nearreal-time data from retail stores 105 to determine or confirm thatretail stores 105 are compliant with the determined planogram ofsupplier 115. As used herein, the term “near real-time data,” in thecontext of this disclosure, refers to data acquired or generated, etc.,based on sensor readings and other inputs (such as data from imagesensors, audio sensors, pressure sensors, checkout stations, etc.) fromretail store 105 received by system 100 within a predefined period oftime (such as time periods having durations of less than a second, lessthan a minute, less than an hour, less than a day, less than a week,etc.).

In some embodiments, server 135 may generate reports that summarizeperformance of the current assortment and the planogram compliance.These reports may advise supplier 115 of the category and the itemperformance based on individual Stock Keeping Unit (SKU), sub segmentsof the category, vendor, and region. In addition, server 135 may providesuggestions or information upon which decisions may be made regardinghow or when to remove markdowns and when to replace underperformingproducts. For example, as shown in FIG. 11B, GUI 1110 may include afirst display area 1112 for showing different scores of supplier 115relative to scores associated with its competitors. GUI 1110 may alsoinclude a second display area 1114 showing the market share of eachcompetitor. GUI 1110 may also include a third display area 1116 showingretail measurements and distribution of brands. GUI 1110 may alsoinclude a fourth display area 1118 showing a suggested planogram. Theprovided information may help supplier 115 to select preferredplanograms based on projected or observed profitability, etc., and toensure that retail stores 105 are following the determined planogram.

FIGS. 11C and 11D illustrate example GUIs for output devices 145C, whichmay be used by store associates of retail store 105. FIG. 11C depicts aGUI 1120 for a manager of retail store 105 designed for a desktopcomputer, and FIG. 11D depicts GUI 1130 and 1140 for store staffdesigned for a handheld device. In-store execution is one of thechallenges retail stores 105 have in creating a positive customerexperience. Typical in-store execution may involve dealing with ongoingservice events, such as a cleaning event, a restocking event, arearrangement event, and more. In some embodiments, system 100 mayimprove in-store execution by providing adequate visibility to ensurethat the right products are located at preferred locations on the shelf.For example, using near real-time data (e.g., captured images of storeshelves) server 135 may generate customized online reports. Storemanagers and regional managers, as well as other stakeholders, mayaccess custom dashboards and online reports to see how in-storeconditions (such as, planogram compliance, promotion compliance, pricecompliance, etc.) are affecting sales. This way, system 100 may enablemanagers of retail stores 105 to stay on top of burning issues acrossthe floor and assign store associates to address issues that maynegatively impact the customer experience.

In some embodiments, server 135 may cause real-time automated alertswhen products are out of shelf (or near out of shelf), when pricing isinaccurate, when intended promotions are absent, and/or when there areissues with planogram compliance, among others. In the example shown inFIG. 11C, GUI 1120 may include a first display area 1122 for showing theaverage scores (for certain metrics) of a specific retail store 105 overa selected period of time. GUI 1120 may also include a second displayarea 1124 for showing a map of the specific retail store 105 withreal-time indications of selected in-store execution events that requireattention, and a third display area 1126 for showing a list of theselected in-store execution events that require attention. In anotherexample, shown in FIG. 11D, GUI 1130 may include a first display area1132 for showing a list of notifications or text messages indicatingselected in-store execution events that require attention. Thenotifications or text messages may include a link to an image (or theimage itself) of the specific aisle with the in-store execution event.In another example, shown in FIG. 11D, GUI 1140 may include a firstdisplay area 1142 for showing a display of a video stream captured byoutput device 145C (e.g., a real-time display or a near real-timedisplay) with augmented markings indicting a status of planogramcompliance for each product (e.g., correct place, misplaced, not inplanogram, empty, and so forth). GUI 1140 may also include a seconddisplay area 1144 for showing a summary of the planogram compliance forall the products identified in the video stream captured by outputdevice 145C. Consistent with the present disclosure, server 135 maygenerate within minutes actionable tasks to improve store execution.These tasks may help store associates of retail store 105 to quicklyaddress situations that can negatively impact revenue and customerexperience in the retail store 105.

FIG. 11E illustrates an example GUI 1150 for output device 145D used byan online customer of retail store 105. Traditional online shoppingsystems present online customers with a list of products. Productsselected for purchase may be placed into a virtual shopping cart untilthe customers complete their virtual shopping trip. Virtual shoppingcarts may be examined at any time, and their contents may be edited ordeleted. However, common problems of traditional online shopping systemsarise when the list of products on the website does not correspond withthe actual products on the shelf. For example, an online customer mayorder a favorite cookie brand without knowing that the cookie brand isout-of-stock. Consistent with some embodiments, system 100 may use imagedata acquired by capturing devices 125 to provide the online customerwith a near real-time display of the retail store and a list of theactual products on the shelf based on near real-time data. In oneembodiment, server 135 may select images without occlusions in the fieldof view (e.g., without other customers, carts, etc.) for the nearreal-time display. In one embodiment, server 135 may blur or erasedepictions of customers and other people from the near real-timedisplay. As used herein, the term “near real-time display,” in thecontext of this disclosure, refers to image data captured in retailstore 105 that was obtained by system 100 within a predefined period oftime (such as less than a second, less than a minute, less than about 30minutes, less than an hour, less than 3 hours, or less than 12 hours)from the time the image data was captured.

Consistent with the present disclosure, the near real-time display ofretail store 105 may be presented to the online customer in a mannerenabling easy virtual navigation in retail store 105. For example, asshown in FIG. 11E, GUI 1150 may include a first display area 1152 forshowing the near real-time display and a second display area 1154 forshowing a product list including products identified in the nearreal-time display. In some embodiments, first display area 1152 mayinclude different GUI features (e.g., tabs 1156) associated withdifferent locations or departments of retail store 105. By selectingeach of the GUI features, the online customer may virtually jump todifferent locations or departments in retail store 105. For example,upon selecting the “bakery” tab, GUI 1150 may present a near real-timedisplay of the bakery of retail store 105. In addition, first displayarea 1152 may include one or more navigational features (e.g., arrows1158A and 1158B) for enabling the online customer to virtually movewithin a selected department and/or virtually walk through retail store105. Server 135 may be configured to update the near real-time displayand the product list upon determining that the online customer wants tovirtually move within retail store 105. For example, after identifying aselection of arrow 1158B, server 135 may present a different section ofthe dairy department and may update the product list accordingly. Inanother example, server 135 may update the near-real time display andthe product list in response to new captured images and new informationreceived from retail store 105. Using GUI 1150, the online customer mayhave the closest shopping experience without actually being in retailstore 105. For example, an online customer may visit the vegetabledepartment and decide not to buy tomatoes after seeing that they are notripe enough.

As explained elsewhere in this disclosure, many retailers and suppliersnowadays send people to stores to personally monitor compliance withdesired product placement in the stores. However, this is inefficientand may result in nonuniform compliance among retailers relative tovarious product-related guidelines. Using cameras to monitor compliancemay help monitor retail spaces more efficiently. The present disclosureprovides systems and methods for planning deployment of one or morecameras within a retail store. In one embodiment, a server may obtain astore plan of a retail store and determine a location of a store shelfwithin the retail store based on the store plan. The server may alsoaccess a database to determine the height of products of a product typethat are placed on the store shelf. The server may further determine acoverage parameter for the product type (e.g., a percentage of thecaptured portion of one product of the product type in an image toenable recognition). The server may also determine a position forplacing a camera configured to capture one or more images of at leastone portion of the store shelf and at least one portion of products ofthe product type placed on the store shelf based on the location of thestore shelf, the height of the products of the product type, and thecoverage parameter.

FIG. 12A is a schematic illustration of an exemplary camera at anexemplary position consistent with some embodiments of the presentdisclosure. As illustrated in FIG. 12A, a store shelves unit within aretail store may include a store shelf 1201 and a store shelf 1202.Store shelf 1202 may have a depth d. The height between store shelf 1201and store shelf 1202 may be referred to as h. Products of a firstproduct type 1221 may be placed on store shelf 1202. Camera 1210 may beplaced at position 1211 such that camera 1210 may have a field of view1212 and be configured to capture an image of at least a portion ofstore shelf 1202 and a portion of the products placed on store shelf1202.

The position of the camera may be determined based on various factors(e.g., the location of a store shelf, one or more properties of productsof one or more product types, one or more camera properties, coverageparameter corresponding to one or more product types, etc.). Forexample, as illustrated in FIG. 12B, products of second product type1222 may be placed on store shelf 1202. Second product type 1222 may behigher than first product type 1221. Accordingly, camera 1230 may beplaced at position 1231, which may be different from position 1211.Camera 1230 may be placed at position 1231 and may have a field of view1232, which may cover more of products of second product type 1222 thanthose when camera 1230 is placed at position 1211. As another example,as illustrated in FIG. 12C, there may be products of two differentproduct types (e.g., first product type 1221 and second product type1222) placed on store shelf 1202. A camera 1240 may be placed atposition 1241, which may have a field of view 1242, to capture at leastone portion of products of first product type 1221 and at least oneportion of products of second product type 1222. For example, inexamples where camera 1240 is placed at position 1211, the imagecaptured by camera 1240 may cover none or few of products of secondproduct type 1222.

FIG. 13 is a block diagram of an exemplary system 1300 for planningdeployment of image sensors. As illustrated in system 1300 may include aserver 1301, user device 1302, one or more cameras 1303 (e.g., 1303A,1303B, . . . , 1303N), a digital communication network 1304, and adatabase 1305. Server 1301 may be configured to determine a position forplacing a camera configured to capture images of a store shelf (or atleast one portion thereof) within a retail store. User device 1302 maybe configured to receive information relating to the position forplacing the camera from server 1301 and display a user interfacepresenting the received information. Additionally or alternatively,system 1300 may include a robot configured to position a camera in theposition determined by server 1301. Camera 1303 may be configured tocapture one or more images at a determined position by server 1301(and/or user device 1302). Digital communication network 1304 may beconfigured to facilitate communications among the components of system1300. Database 1305 may be configured to store data that may be accessedby one or more components of system 1300.

In some embodiments, server 1301 may be configured to determine aposition for placing a camera (e.g., camera 1303) configured to captureimages of at least a portion of a store shelf in a retail store. Forexample, server 1301 may be configured to determine a position forplacing camera 1303 based on the location of the store shelf, a firstcoverage parameter corresponding to a first product type, a secondcoverage parameter corresponding to a second product type, a firstheight of products of the first product type, and a second height ofproducts of the second product type. Server 1301 may also be configuredto transmit information relating to the determined position of thecamera to user device 1302 or to another external device. Server 1301may include at least one processor configured to perform one or morefunctions thereof. Server 1301 may also include memory configured tostore instructions for the at least one processor. Server 1301 mayfurther include at least one storage device configured to store data forthe at least one processor.

In some embodiments, user device 1302 may be configured to presentinformation and receive user input via a user interface. For example,user device 1302 may receive information relating to a position forplacing the camera from server 1301. User device 1302 may also beconfigured to present the information relating to the position forplacing the camera in a user interface. In some other examples, userdevice 1302 may be configured to provide the information relating to theposition for placing the camera in an audible output, in a textualoutput, in a graphical output, in an overlay over an image, in anaugmented reality system (for example by providing a visual indicationof the position in the augmented reality system), or the like, or acombination thereof. In some embodiments, user device 1302 may beconfigured to receive input from the user via the user interface (and/oran input device associated with user device 1302). For example, userdevice 1302 may be configured to receive user input for modifying theposition of the camera. User device 1302 may include at least oneprocessor configured to perform one or more functions thereof. Userdevice 1302 may also include memory configured to store instructions forthe at least one processor. User device 1302 may further include atleast one storage device configured to store data for the at least oneprocessor. User device 1302 may include a mobile computing device, apersonal computing device of a user, a mobile communication device, apersonal communication device of a user, a smartphone, a tablet, apersonal computer, a virtual reality system, an augmented realitysystem, or the like, or a combination thereof.

In some embodiments, at least one processor of server 1301 and/or userdevice 1302 may include a microprocessor, preprocessors (such as animage preprocessor), a graphics processing unit (GPU), a centralprocessing unit (CPU), support circuits, digital signal processors,integrated circuits, memory, or any other types of devices suitable forrunning applications or performing a computing task. In someembodiments, the at least one processor may include any type of singleor multi-core processor, mobile device microcontroller, centralprocessing unit, etc. Various processing devices may be used, including,for example, processors available from manufacturers such as Intel®,AMD®, etc., or GPUs available from manufacturers such as NVIDIA®, ATI®,etc. and may include various architectures (e.g., x86 processor, ARM®,etc.). In other examples, server 1301 may be part of or implemented by acloud platform. Any of the processing devices disclosed herein may beconfigured to perform certain functions. Configuring a processingdevice, such as any of the described processors or other controller ormicroprocessor, to perform certain functions may include programming ofcomputer-executable instructions and making those instructions availableto the processing device for execution during operation of theprocessing device. In some embodiments, configuring a processing devicemay include programming the processing device directly witharchitectural instructions. For example, processing devices such asfield-programmable gate arrays (FPGAs), application-specific integratedcircuits (ASICs), and the like may be configured using, for example, oneor more hardware description languages (HDLs).

In some embodiments, camera 1303 may be configured to capture one ormore images of at least a portion of a store shelf and/or one or moreproducts placed on a store shelf. Camera 1303 may include a digitalcamera, a color camera, a time-of-flight camera, a stereo camera, anactive stereo camera, a depth camera, a Lidar system, a laser scanner,CCD based devices, or any other sensor based system capable ofconverting received light into electric signals. In some embodiments,camera 1303 may be configured to capture one or more images based on oneor more capturing parameters configured by server 1301 and/or userdevice 1302. For example, server 1301 may transmit a capturing frequency(e.g., one image per hour) to camera 1303, which may be configured tocapture images based on the capturing frequency. In some embodiments,camera 1303 may be configured to transmit one or more images to server1301, user device 1302, and/or database 1305 via network 1304.

Digital communication network 1304 may be a public network or privatenetwork and may include, for example, a wired or wireless network,including, without limitation, a Local Area Network (LAN), a Wide AreaNetwork (WAN), a Metropolitan Area Network, an IEEE 802.11 wirelessnetwork (e.g., “Wi-Fi”), a network of networks (e.g., the Internet), aland-line telephone network, or the like. Digital communication network1304 may be connected to other networks (not depicted in FIG. 13) toconnect the various system components to each other and/or to externalsystems or devices. In some embodiments, digital communication network1304 may be a secure network and require a password to access thenetwork.

Database 1305 may store information and data for the components ofsystem 1300 (e.g., server 1301, user devices 1302, and/or one or morecameras 1303). In some embodiments, server 1301, user devices 1302,and/or one or more cameras 1303 may be configured to access database1305, and obtain data stored from and/or upload data to database 1305via digital communication network 1304.

Database 1305 may include a cloud-based database or an on-premisesdatabase. Database 1305 may include images captured by one or morecameras 1303, simulated images generated by server 1301 and/or userdevice 1302, configuration data, expression data, datasets, model data(e.g., model parameters, training criteria, performance metrics, etc.),and/or other data, consistent with disclosed embodiments.

FIG. 14 illustrates a flowchart of an exemplary process 1400 fordetermining a position for placing a camera consistent with someembodiments of the present disclosure.

At step 1401, at least one processor of server 1301 may determine alocation of a store shelf within a retail store. The location of a storeshelf may include the position of the store shelf in a store map, in aplan of the retail store, in a model of the retail store, in acoordinate system, or the like, or a combination thereof. Alternativelyor additionally, the location of a store shelf may include the positionof the store shelf in a larger component, for example, in a shelvingunit, in a display unit, in an aisle, in a section of the store, or thelike, or a combination thereof. Alternatively or additionally, thelocation of a store shelf may include the position of the store shelfrelative to an object (e.g., a product, a door of the retail shelf,another store shelf, and so forth) or relative to a known position(e.g., to an entrance to the retail store, to an origin of a coordinatesystem, an aisle, or the like). For example, the location of the storeshelf may include a distance (e.g., a height) of the store shelfrelative to an object. By way of example, the height of the store shelfmay include a height of the store shelf relative to the ground, theceiling, another store shelf (e.g., a store shelf below or above thestore shelf, a store shelf on the other side of the aisle), a side of astore shelves unit (e.g., the top, the bottom, etc.), a product of afirst product type, or the like. In some embodiments, the store shelfmay be indicated to hold (or support) one or more particular types ofproducts (e.g., sodas, cereals, etc.). As another example, the locationof the store shelf may be indicated by a direction (e.g., above, below,opposed, etc.) relative to another store shelf, relative to a fixture ina retail store (e.g., a lighting fixture), a cash register, a check-outlane, a particular product type, or the like, or a combination thereof.For example, the location of the store shelf may be above (or below)products of the first product type and products of the second producttype or across an aisle from products of the first product type andproducts of the second product type.

In some embodiments, the at least one processor may obtain a store planor a store map, and determine the location of the store shelf based onthe store plan. A store plan may include a planogram, a realogram, athree-dimensional (3D) model of the retail store, or the like, or acombination thereof. A realogram may be a virtual copy of shelves in astore. In some embodiments, the store plan may be determined based on ananalysis of images captured from the retail store to identify storeshelves and products placed on the store shelves. For example, one ormore cameras may be configured to capture images of one or more storeshelves and products placed on the store shelves (e.g., images capturedusing a 3D camera (such as a stereo camera, an active stereo camera, atime-of-flight camera, a LiDAR camera, etc.)) The images may be used toconstruct a store plan representing at least one product characteristicrelative to a display structure associated with a retail environment(such as a store shelf or area of one or more shelves). Exemplaryproduct characteristics may include quantities of products with respectto areas of the shelves, product configurations or product shapes withrespect to areas of the shelves, product arrangements with respect toareas of the shelves, product density with respect to areas of theshelves, product combinations with respect to areas of the shelves, etc.The at least one processor may determine a location of a particularstore shelf based on the store plan (e.g., a height of the store shelfrelative to products of a product type).

At step 1402, at least one processor of server 1301 may obtain a firstcoverage parameter corresponding to a first product type and a secondcoverage parameter corresponding to a second product type. The coverageparameter corresponding to a product type may include at least one of adepth within a shelf on which a plurality of products of the producttype are planned to be placed, a percentage of the plurality of productsof the product type that have been to be captured in an image to enablerecognition, a percentage of the captured portion of one product in animage to enable recognition, a capturing resolution, or a capturingfrequency. By way of example, referring to FIG. 12C, the at least oneprocessor may be configured to determine a first coverage parametercorresponding to first product type 1221 and a second coverage parametercorresponding to second product type 1222. The first coverage parametercorresponding to first product type 1221 may include at least one of afirst depth within a shelf on which a plurality of products of firstproduct type 1221 are planned to be placed, a first percentage of theplurality of products of first product type 1221 that have been to becaptured in an image to enable recognition, a first capturingresolution, or a second capturing frequency. The second coverageparameter corresponding to second product type 1222 may include at leastone of a second depth within a shelf on which a plurality of products ofsecond product type 1222 are planned to be placed, a second percentageof the plurality of products of second product type 1222 that have beento be captured in an image to enable recognition, a second capturingresolution, or a second capturing frequency. The first depth may be thesame as or different from the second depth. The first percent may be thesame as or different from the second percent. The first capturingresolution may be the same as or different from the second capturingresolution. The first capturing frequency may be the same as ordifferent from the second capturing frequency.

In some embodiments, the at least one processor may determine a producttype and determine a coverage parameter based on the product type. Forexample, the at least one processor may determine a particular producttype based on a store plan, a store map, a planogram, a realogram, userinput, one or more images of the store shelf supporting the products ofthe product type, or the like, or a combination thereof. The at leastone processor may also determine a coverage parameter based on theproduct type by, for example, consulting a lookup table including aplurality of coverage parameters corresponding to a plurality of producttypes.

At step 1403, at least one processor of server 1301 may access adatabase to determine a first height of products of the first producttype and a second height of products of the second product type. By wayof example, referring to FIG. 12C, the at least one processor may accessa database (e.g., a database in a local store device, database 1305,etc.), which may include the properties of one or more product types(e.g., the size, one or more dimensions, the weight, the volume, etc.).The at least one processor may also determine a first height of productsof first product type 1221 and a second height of products of secondproduct type 1222. In addition to or alternative to accessing thedatabase, the at least one processor may analyze images of products ofthe first product type to determine the height of products of the firstproduct type and/or may images of products of the second product type todetermine the height of products of the second product type.

At step 1404, at least one processor of server 1301 may determine aposition for placing a camera configured to capture images of at least aportion of the store shelf by analyzing the location of the store shelf,the first coverage parameter, the second coverage parameter, the firstheight, and the second height. By way of example, referring to FIG. 12C,as described above, the at least one processor may determine thelocation of store shelf 1202 (e.g., the height of store shelf 1202relative to store shelf 1201), the first coverage parametercorresponding to first product type 1221, the second coverage parametercorresponding to second product type 1222, the first height of firstproduct type 1221, and the second height of second product type 1222.The at least one processor may also determine position 1241 for placingcamera 1240 configured images of at least a portion of store shelf 1202based on the location of store shelf 1202, the first coverage parameter,the second coverage parameter, the first height, and the second height.In some embodiments, camera 1240 may be positioned such that one or moreimages captured by camera 1240 also include at least a portion of oneproduct of first product type 1221 and at least a portion of one productof second product type 1222. In some embodiments, a machine learningmodel may be trained using training samples to determine positions forplacing cameras from locations of the store shelves, coverageparameters, and/or heights of products. For example, a training examplemay include a sample location of a sample store shelf, a sample firstcoverage parameter, a sample second coverage parameter, a sample firstheight, a sample second height, and/or a label indicating a desiredposition for placing a sample camera. The at least one processor may usethe trained machine learning model to analyze the location of the storeshelf, the first coverage parameter, the second coverage parameter,and/or the first height and the second height to determine the positionfor placing the camera configured to capture images of at least theportion of the store shelf. Alternatively or additionally, the at leastone processor may use an optimization algorithm to determine a positionfor placing the camera that maximizes a criterion function of a coverageof the camera while ensuring one or more desired minimal requirements.

In some embodiments, the at least one processor may also determine anorientation (or a pointing direction) of the camera. For example, the atleast one processor may determine an orientation (or a pointingdirection) of the camera base on the location of the store shelf, thefirst coverage parameter, the second coverage parameter, the firstheight, and the second height, such that at least a portion of oneproduct of first product type 1221 and at least a portion of one productof second product type 1222 is within the field of view of camera 1240(e.g., field of view 1242). In some embodiments, a machine learningmodel may be trained using training samples to determine the orientationof a camera from the location of a store shelf, one or more coverageparameters, and/or the height(s) of one or more products. A trainingsample may include a sample location of a sample store shelf, a samplefirst coverage parameter, a sample second coverage parameter, a samplefirst height, a sample second height, and/or a label indicating adesired orientation for a sample camera. The at least one processor mayuse the trained machine learning model to analyze the location of thestore shelf, the first coverage parameter, the second coverageparameter, and/or the first height and the second height to determinethe orientation of the camera configured to capture images of at leastthe portion of the store shelf. Alternatively or additionally, the atleast one processor may use an optimization algorithm to determine anorientation the camera that maximizes a criterion function of a coverageof the camera while ensuring one or more desired minimal requirements.

In some embodiments, the at least one processor may determine acapturing frequency for the camera. The capturing frequency for thecamera may include a frequency at which the camera is configured tocapture an image of the store shelf (or a portion thereof) at thedetermined position. In some embodiments, the capturing frequency forthe camera may be different from the maximal capturing frequency of thecamera. For example, the camera may be used to capture one image of thestore shelf (or a portion thereof) per hour, although the camera can beconfigured to capture six images per second at the maximum speed. The atleast one processor may determine a capturing frequency for the camerabased on at least one of the location of the store shelf, the firstcoverage parameter, the second coverage parameter, the first height, thesecond height, the first product type, the second product type, or thelike, or a combination thereof. For example, the first product type andthe second product type may be products that are taken off the storeshelf by customers frequently. The at least one processor may determinea capturing frequency for the camera for these product types that may behigher than the capturing frequency for a product type that is lessfrequently taken off the store shelf.

In some embodiments, in determining the position for placing the camera,the at least one processor may take other factors into account. Forexample, the at least one processor may determine at least one dimensionof the store shelf (e.g., a depth of the store shelf, a length of thestore shelf, a thickness of the store shelf, etc.). The at least oneprocessor may also determine the position for placing the cameraconfigured to capture images of at least the portion of the store shelfby analyzing the location of the store shelf, the at least one dimensionof the store shelf, the first coverage parameter, the second coverageparameter, the first height, and the second height. Alternatively oradditionally, the at least one processor may determine (or receive) atleast one property of the camera. Exemplary properties may include oneor more pixel resolutions, a sensor dimension, a focal length, a rangeof focal lengths (e.g., for a camera having an adjustable focal length),an angle-of-view, a rotation range, a mounting constraints of the camera(e.g., whether it can be mounted to a shelf), a power requirement (e.g.,requiring an external power supply or not), or the like, or acombination thereof. The at least one processor may determine theposition for placing the camera based on the at least one property ofthe camera in addition to other factors described in this disclosure.Alternatively or additionally, in determining the position for placingthe camera, the at least one processor may take another store shelfand/or the products placed thereon into account. For example, the atleast one processor may determine the location of a second store shelfwithin the retail store (similar to step 1401). The second store shelfmay be adjacent to the first store shelf (i.e., the store shelfreferenced in connection with step 1401). Alternatively, the secondstore shelf may be above or below the first store shelf. For example,the second store shelf may be positioned above the products of the firstproduct type and the products of the second product type and/or abovethe store shelf, which may be indicated by, for example, a store plan, astore map, a planogram, a realogram, an image of the first and secondstore shelves at the current state, or the like, or a combinationthereof. The at least one processor may determine the position forplacing the camera configured to capture images of at least the portionof the store shelf by analyzing the location of the first store shelf,the location of the second store shelf, the first coverage parameter,the second coverage parameter, the first height, and the second height.

At step 1405, at least one processor of server 1301 may provide, to auser interface of a user device, information relating to the determinedposition of the camera. For example, referring to FIG. 12C, the at leastone processor may provide information relating to the determinedposition 1241 of camera 1240 to a user interface of user device 1302. Byway of example, the provided information may include an indication of atleast a portion of the store shelf in a field of view of the camera atthe determined position (e.g., information relating to a particularproduct type corresponding to the at least portion of the store shelf).In some embodiments, the information relating to the determined positionof the camera is provided through an augmented reality (AR) system. Forexample, user device 1302 (or an AR system associated with user device1302) may display the camera (or a text or graphical representationthereof) at the determined position overlapping with the images capturedfrom the environment of the store shelf. In some embodiments, theprovided information may include an indication of a particular storeshelf to which the camera is to be mounted at or near the determinedposition. For example, the at least one processor may determine a storeshelf that is on the other side of the aisle and can mount the camera ator near the determined position (which may also be referred to herein asthe mounting store shelf). The at least one processor may also cause theuser interface of the user device to indicate (and/or display) themounting store shelf. In some embodiments, the indication of aparticular store shelf may also include information relating to one ormore product types corresponding to the particular store shelf (e.g.,one or more product types placed on, below, and/or above the particularstore shelf).

In some embodiments, the at least one processor may be programmed toselect a camera among a plurality of cameras as the camera configured tocapture images of the at least a portion of the store shelf based onproperties of each one of the plurality of alternative cameras, thelocation of the store shelf, the first coverage parameter, the secondcoverage parameter, the first height, and the second height. Theselected camera may be different from at least one of the plurality ofcameras in at least one property (one or more pixel resolutions, asensor dimension, a focal length, a range of focal lengths (e.g., for acamera having an adjustable focal length), an angle-of-view, a rotationrange, a mounting constraints of the camera (e.g., whether it can bemounted to a shelf), a power requirement (e.g., requiring an externalpower supply or not), or the like, or a combination thereof). Forexample, the at least one processor may select a camera having thegreatest field of view among a plurality of cameras based on propertiesof each one of the plurality of alternative cameras, the location of thestore shelf, the first coverage parameter, the second coverageparameter, the first height, and the second height. The at least oneprocessor may also be programmed to provide an indication of theselected camera to the user via, for example, the user interface of userdevice 1302.

In some embodiments, the at least one processor may be programmed togenerate a simulated image corresponding to a field of view of thecamera. For example, the at least one processor may generate a simulatedimage that may resemble an image captured by the camera at thedetermined position corresponding to the field of view of the camera. Byway of example, the at least one processor may generate a simulatedimage using a machine learning model (e.g., a model trained using aGenerative Adversarial Network (GAN)). The at least one processor mayenter various parameters (e.g., at least one of a camera property, theposition of the camera, or the orientation of the camera, etc.) as inputto the machine learning model, which may generate a simulated image asthe output. In some embodiments, a simulated image may includeillustrations of a least a portion of the store shelf, a first pluralityof products of the first product type positioned on the store shelf, anda second plurality of products of the second product type positioned onthe store shelf.

In some embodiments, the at least one processor may receive an updatedposition of the camera and determine an updated simulated image based onthe updated position of the camera. For example, as described elsewherein this disclosure, user device 1302 may display in a user interface anindication of the camera at a determined position. The user may modifythe position of the camera via the user interface of user device 1302,which may transmit to server 1301 the user input for modifying theposition of the camera (or data relating to the modified position of thecamera). The at least one processor of server 1301 may generate anupdated simulated image corresponding to the modified position of thecamera.

In some embodiments, the at least one processor may update the positionfor placing the camera based on updated information relating to thestore shelf. For example, the location of the store shelf may change dueto an update of the position of the store shelf immediately above thestore shelf. The at least one processor may receive the updatedinformation, determine an updated to at least one of a position of thecamera and/or an orientation of the camera based on the updatedinformation, and provide information relating to the updated to at leastone of the position of the camera and the orientation of the camera tothe user interface of user device 1302 (similar to process 1400described above). Alternatively or additionally, the at least oneprocessor may update the position for placing the camera and/or theorientation of the camera based on an update in a product typeassociated with the store shelf. For example, the at least one processormay receive updated information relating to a product type associatedwith products placed on the store shelf (e.g., the first product type,the second product type, and/or another product type). By way ofexample, the second product type may not be placed on the store shelf,and products of a third product type may be placed on the store shelf.As another example, the placement of products of the first product typeand products of the second product type may change (e.g., products ofthe first product type may occupy a larger area than before). The atleast one processor may receive the updated information. The at leastone processor may also be programmed to determine an updated position ofthe camera based on the updated information and provide an indication ofthe updated position in the user interface of user device 1302 (similarto process 1400 described above). Alternatively or additionally, the atleast one processor may be programmed to determine an updatedorientation of the camera at the originally determined position (similarto process 1400 described above).

While the description of process 1400 is provided using server 1301 asan example, one having ordinary skill in art would understand that userdevice 1302 may be configured to perform one or more steps of process1400.

The present disclosure provides systems and methods for navigating oneor more cleaning robots to capturing images of retail store shelves,which may monitor retail spaces more efficiently. In one embodiment, aserver may receive a request for checking a store shelf in a retailstore and direct a cleaning robot (or a vehicle) to a position at whichan image sensor (e.g., a camera) associated with the cleaning robot maybe configured to capture one or more images of at least one portion ofthe store shelf. The cleaning robot may transmit at least one capturedimage to the server via, for example, one or more networks. The servermay be configured to analyze the image received from the cleaning robotto determine a need for another image of the store shelf (or a portionthereof). For example, the server may analyze the image and determinethat the image does not capture a sufficient portion of the store shelfto recognize the product items placed on the store shelf, or that theimage does not capture a particular portion of the store shelf at asufficient quality (e.g., having an insufficient resolution, aninsufficient sharpness, an inadequate angle, etc.) to recognize theproduct items placed on the particular portion of the store shelf. Theserver may direct the cleaning robot to move to a different position(and/or change the orientation of the camera) at which the camera may beconfigured to capture one or more images of at least a portion of thestore shelf. The cleaning robot may also be configured to transmit thenewly captured image(s) to the server.

FIG. 15 illustrates an exemplary process for navigating a cleaning robot1501. As illustrated in FIG. 15, cleaning robot 1501 may initiallynavigate along a route 1511 through aisle 1521, aisle 1522, and aisle1523, to clean areas along route 1511. In some embodiments, cleaningrobot 1501 may also include a camera 1502 configured to capture one ormore images from the environment of cleaning robot 1501, including, forexample, at least one portion of a store shelf and/or one or moreproduct items. In some embodiments, cleaning robot 1501 may include aplurality of cameras 1502 (e.g., two, three, four, five, six, etc.). Forexample, in some embodiments, cleaning robot 1501 may include aplurality of cameras 1502 to capture views facing in front, behind,and/or one or more sides of cleaning robot 1501. As another example, insome embodiments, one or more cameras 1502 included in cleaning robot1501 may be configured to rotate in order to focus on a particular areawithout changing a facing direction of cleaning robot 1501. Cleaningrobot 1501 may receive information relating to route 1511 from a serverand/or a user device. The server (and/or a computing device external orinternal to cleaning robot 1501) may receive an indication of storeshelf 1531 and cause a first adjustment to route 1511 based on theindication of store shelf 1531. For example, the server may determinethe first adjustment to route 1511 to cover store shelf 1531 or to covera position in proximity of store shelf 1531. By way of example, asillustrated in FIG. 15, the server may determine a modified route 1512,which may include a detour to store shelf 1531 along aisle 1524 and thento aisle 1523. The server may also transmit the first adjustment toroute 1511 (and/or modified route 1512) to cleaning robot 1501. Cleaningrobot 1501 may follow modified route 1512 instead of original route1511.

FIG. 16 illustrates exemplary cleaning robot 1501 and exemplary storeshelves 1531, 1532, and 1533. As described above, the server (and/or auser device) may direct cleaning robot 1501 to move to a position closeto store shelf 1531 according to the first adjustment to cleaning robot1501's route. Cleaning robot 1501 may arrive at a position in proximityof store shelf 1531. Camera 1502, which may be disposed on cleaningrobot 1501, may be configured to capture one or more images of at leastportion of store shelf 1531. For example, as illustrated in FIG. 16,camera 1502 may be configured to capture one or more images of the itemswithin FOV 1631, which may include at least a portion of store shelf1531 and one or more items of first product type 1621.

In some embodiments, camera 1502 and/or cleaning robot 1501 may beconfigured to change camera 1502's orientation, position, and/or fieldof view. For example, camera 1502 (and/or cleaning robot 1501) may beconfigured to change its orientation such that its field of view maychange to FOV 1632 shown in FIG. 16 through one or more pan/tiltoperations. Camera 1502 may be configured to capture one or more imagesof objects within FOV 1632, which may include representations of atleast a portion of one or more items of first product type 1621, atleast a portion of store shelf 1532, at least a portion of one or moreitems of second product type 1622, at least a portion of store shelf1532, and at least a portion of one or more items of second product type1623. Alternatively or additionally, camera 1502 and/or cleaning robot1501 may be configured to change the height of camera 1502 such thatcamera 1502 may be configured to capture one or more images at a higherposition (e.g., to capture one or more images of items placed on storeshelf 1533 at a height close to the horizontal level of store shelf1532).

FIG. 17A is a block diagram of an example system 1700 for navigating acleaning robot. As illustrated in FIG. 17A, system 1700 may include aserver 1701, user device 1702, one or more cleaning robots 1501 (e.g.,1501A, 1501B, . . . , 1501N), a network 1704, and a database 1705.Server 1701 may be configured to determine a route and/or one or moreadjustments to a route for cleaning robot 1501. In one example, server1701 may be external to cleaning robots 1501. In another example, server1701 may be internal to at least one of cleaning robots 1501. In yetanother example, server 1701 may be internal to user device 1702. In anadditional example, server 1701 may be external to user device 1702.User device 1702 may be configured to receive information relating toone or more store shelves and/or cleaning robot 1501, and present thereceived information to one or more individuals (e.g., a worker, a storeassociate, etc.). For example, user device 1702 may receive a requestfor restocking items associated with a store shelf from server 1701and/or cleaning robot 1501. Cleaning robot 1501 may be configured toclean a surface along a route in the retail store. In some embodiments,cleaning robot 1501 may include a camera configured to capture one ormore images from the environment of cleaning robot 1501. Network 1704may be configured to facilitate communications among the components ofsystem 1700. Database 1705 may be configured to store data that may beaccessed by one or more components of system 1700.

In some embodiments, server 1701 and/or user device 1702 may beconfigured to receive a first image acquired by the camera associatedwith cleaning robot 1501. The first image may include a representationof at least one portion of a particular store shelf. Server 1701 and/oruser device 1702 may also be configured to analyze the first image todetermine a need for a second image of the at least one portion of theat least one store shelf. Server 1701 and/or user device 1702 mayfurther be configured to cause a second adjustment to the route ofcleaning robot 1501 within the retail store in response to thedetermined need.

Server 1701 may include at least one processor configured to perform oneor more functions thereof. Server 1701 may also include memoryconfigured to store instructions for the at least one processor. Server1701 may further include at least one storage device configured to storedata for the at least one processor.

In some embodiments, user device 1702 may be configured to presentinformation and receive user input via a user interface. For example,user device 1702 may receive an instruction to restock items associatedwith a particular store shelf. User device 1702 may also be configuredto present the instruction to the user in a user interface. In someembodiments, user device 1702 may be configured to receive input fromthe user via the user interface (and/or an input device associated withuser device 1702). For example, user device 1702 may be configured toreceive user input to confirm that a restocking task associated with astore shelf has been completed. User device 1702 may also transmit anotification indicating the completed restocking task to server 1701 vianetwork 1704. In one example, user device 1702 may receive at least partof an image captured using camera 1502 and/or cleaning robot 1501, andmay present information based on the received at least part of an image(for example, presenting the at least part of the image, informationbased on an analysis of the at least part of the image, or the like, ora combination thereof).

User device 1702 may include at least one processor configured toperform one or more functions thereof. User device 1702 may also includememory configured to store instructions for the at least one processor.User device 1702 may further include at least one storage deviceconfigured to store data for the at least one processor.

In some embodiments, at least one processor of server 1701 and/or userdevice 1702 may include a microprocessor, preprocessors (such as animage preprocessor), a graphics processing unit (GPU), a centralprocessing unit (CPU), support circuits, digital signal processors,integrated circuits, memory, or any other types of devices suitable forrunning applications or performing a computing task.

In some embodiments, the at least one processor may include any type ofsingle or multi-core processor, mobile device microcontroller, centralprocessing unit, etc. Various processing devices may be used, including,for example, processors available from manufacturers such as Intel®,AMD®, etc., or GPUs available from manufacturers such as NVIDIA®, ATI®,etc. and may include various architectures (e.g., x86 processor, ARM®,etc.). Any of the processing devices disclosed herein may be configuredto perform certain functions. Configuring a processing device, such asany of the described processors or other controller or microprocessor,to perform certain functions may include programming ofcomputer-executable instructions and making those instructions availableto the processing device for execution during operation of theprocessing device. In some embodiments, configuring a processing devicemay include programming the processing device directly witharchitectural instructions. For example, processing devices such asfield-programmable gate arrays (FPGAs), application-specific integratedcircuits (ASICs), and the like may be configured using, for example, oneor more hardware description languages (HDLs).

In some embodiments, cleaning robot 1501 may be configured to clean,sweep, and/or scrub a surface autonomously or semi-autonomously.Cleaning robot 1501 may include an electronics system configured toperform and/or execute a set of instructions to control at least one ofa drive system, a cleaning assembly, a vacuum source, a pump, a motor,or the like, or a combination thereof based on one or more signalsassociated with an operational condition of cleaning robot 1501 and/orone or more environmental conditions associated with the area to becleaned. In some embodiments, cleaning robot 1501 may include one ormore sensors configured to determine its position (e.g., a globalposition or a local position relative to an object) and/or detect one ormore environment conditions. For example, cleaning robot 1501 mayinclude one or more of a camera, a light emitting and/or sensing device(e.g., visible light, infrared light, etc.), a radio and/or sound waveemitter (e.g., sonar), a global positioning system (GPS) device, aproximity sensor, a LIDAR device, or the like, or a combination thereof.

In some embodiments, a camera associated with cleaning robot 1501 may beconfigured to capture one or more images from the environment ofcleaning robot 1501. For example, the camera may be configured tocapture one or more images including a representation of at least aportion of a store shelf and/or one or more products placed on a storeshelf. The camera may include a digital camera, a time-of-flight camera,a stereo camera, an active stereo camera, a depth camera, a Lidarsystem, a laser scanner, CCD based devices, or any other sensor basedsystem capable of converting received light into electric signals. Insome embodiments, the camera may be configured to capture one or moreimages based on one or more capturing parameters configured by server1701 and/or user device 1702. For example, server 1701 may transmit tothe camera a resolution parameter specifying a resolution at which animage to be captured. The camera may be configured to capture one ormore images at the specified resolution. In some embodiments, the cameramay be configured to transmit one or more images to server 1701, userdevice 1702, and/or database 1705 via network 1704 (or through cleaningrobot 1501).

Network 1704 may be a public network or private network and may include,for example, a wired or wireless network, including, without limitation,a Local Area Network (LAN), a Wide Area Network (WAN), a MetropolitanArea Network, an IEEE 802.11 wireless network (e.g., “Wi-Fi”), a networkof networks (e.g., the Internet), a land-line telephone network, or thelike. Network 1704 may be connected to other networks (not depicted inFIG. 17) to connect the various system components to each other and/orto external systems or devices. In some embodiments, network 1704 may bea secure network and require a password to access the network.

Database 1705 may store information and data for the components ofsystem 1700 (e.g., server 1701, user devices 1702, one or more cleaningrobots 1501, and/or one or more cameras associated with cleaning robots1501). In some embodiments, server 1701, user devices 1702, one or morecleaning robots 1501, and/or one or more cameras associated withcleaning robots 1501 may be configured to access database 1705, andobtain data stored from and/or upload data to database 1705 via network1704. Database 1705 may include a cloud-based database or an on-premisesdatabase. Database 1705 may include images captured by one or morecameras, one or more routes for cleaning robot 1501, configuration data,expression data, datasets, model data (e.g., model parameters, trainingcriteria, performance metrics, etc.), and/or other data, consistent withdisclosed embodiments.

FIG. 17B illustrates a flowchart of an exemplary process 1710 fornavigating cleaning robot 1501 consistent with some embodiments of thepresent disclosure.

At step 1711, at least one processor of server 1701 may be programmed toreceive an indication of at least one store shelf of a retail store. Anindication of a store shelf may include at least one of an identity ofthe store shelf (e.g., a shelf number, an aisle number of the aisleassociated with the store shelf, etc.), an identity of the shelves unitwhere the store shelf is located, the location of the store shelf in theretail store, the location of the shelving unit, or the like, or acombination thereof. Alternatively or additionally, an indication of astore shelf may include a request for checking a particular store shelfand/or an area associated with the store shelf. For example, server 1701may receive a request from a worker for checking one or more itemsplaced on a store shelf, which may include an indication of the storeshelf. As another example, server 1701 may receive a request from aworker or a customer to clean an area, which may be associated with oneor more store shelves.

In some embodiments, the location of a store shelf may include theposition of the store shelf relative to an object (e.g., a product). Forexample, the location of the store shelf may include a distance (e.g., aheight) of the store shelf relative to an object. By way of example, theheight of the store shelf may include a height of the store shelfrelative to the ground, the ceiling, another store shelf (e.g., a storeshelf below or above the store shelf, a store shelf on the other side ofthe aisle), a side of a store shelves unit (e.g., the top, the bottom,etc.), a product of a first product type, or the like. In someembodiments, the store shelf may be indicated to hold (or support) oneor more particular types of products (e.g., sodas, cereals, etc.). Asanother example, the location of the store shelf may be indicated by adirection (e.g., above, below, opposed, etc.) relative to another storeshelf, relative to a fixture in a retail store (e.g., a lightingfixture), a cash register, a check-out lane, a particular product type,or the like, or a combination thereof. For example, the location of thestore shelf may be above (or below) products of the first product typeand products of the second product type or across an aisle from productsof the first product type and products of the second product type.

In some embodiments, server 1701 may determine a product type (e.g., asoda) and one or more store shelves associated with the product type.For example, server 1701 may receive a request for checking the stockingof a particular soda and access a database (e.g., database 1705) toidentify a store shelf associated with the soda. In some embodiments,server 1701 may also determine the location of the store shelf asdescribed elsewhere in this disclosure.

In some embodiments, server 1701 may receive an indication of two ormore store shelves (e.g., store shelf 1531, store shelf 1532, storeshelf 1533 illustrated in FIG. 16). Server 1701 may also determine thelocations of the store shelves.

In some embodiments, the at least one processor may obtain a store planor a store map, and determine the location of the store shelf based onthe store plan. A store plan may include a planogram, a realogram, athree-dimensional (3D) model of the retail store, or the like, or acombination thereof. A realogram may be a virtual copy of shelves in astore. In some embodiments, the store plan may be determined based on ananalysis of images captured from the retail store to identify storeshelves and products placed on the store shelves. For example, one ormore cameras may be configured to capture images of one or more storeshelves and products placed on the store shelves (e.g., images capturedusing a 3D camera (such as a stereo camera, an active stereo camera, atime-of-flight camera, a LiDAR camera, etc.)). The images may be used toconstruct a store plan representing at least one product characteristicrelative to a display structure associated with a retail environment(such as a store shelf or area of one or more shelves). Exemplaryproduct characteristics may include quantities of products with respectto areas of the shelves, product configurations or product shapes withrespect to areas of the shelves, product arrangements with respect toareas of the shelves, product density with respect to areas of theshelves, product combinations with respect to areas of the shelves, etc.The at least one processor may determine a location of a particularstore shelf based on the store plan (e.g., a height of the store shelfrelative to products of a product type).

At step 1712, at least one processor of server 1701 may cause a firstadjustment to a route of cleaning robot 1501 within the retail storebased on at least one location within the retail store corresponding tothe at least one store shelf. The first adjustment may be configured toenable a camera (e.g., camera 1502) associated with cleaning robot 1501to capture one or more images of at least one portion of the at leastone store shelf. For example, as illustrated in FIG. 15, cleaning robot1501 may initially navigate along a route 1511 through aisle 1521, aisle1522, and aisle 1523, to clean areas along route 1511. Cleaning robot1501 may receive information relating to route 1511 from server 1701and/or user device 1702. The server may determine the first adjustmentto route 1511 to cover store shelf 1531 or to a position in proximity ofstore shelf 1531. By way of example, as illustrated in FIG. 15, theserver may determine a modified route 1512, which may include a detourto store shelf 1531 along aisle 1524 and then to aisle 1523. The servermay also transmit the first adjustment to route 1511 (and/or modifiedroute 1512) to cleaning robot 1501. Cleaning robot 1501 may followmodified route 1512 instead of original route 1511. In some embodiments,the cleaning robot route prior to the first adjustment (e.g., route1511) may not include a path passing the at least one store shelf, andthe cleaning robot route after the first adjustment (e.g., route 1512)may include a path passing the at least one store shelf. In anotherexample, the cleaning robot route prior to the first adjustment mayinclude a path passing the at least one store shelf at a first portionof the route, and the cleaning robot route after the first adjustmentmay include a path passing the at least one store shelf at a secondportion of the route, and the cleaning robot may pass the second portionof the route before passing the first portion of the route. In someembodiments, before the first adjustment to the route, cleaning robot1501 may be configured not to clean a particular portion of the retailstore (such as aisle 1524), and after (and/or as a result of) the firstadjustment to the route, cleaning robot 1501 may be configured to cleanthe particular portion of the retail store. In one example, server 1701may be further configured to cause an adjustment to a route of a secondcleaning robot, for example to exclude the particular portion of theretail store from the route of the second cleaning robot. In anotherexample, server 1701 may be further configured to provide instructionsto a second cleaning robot, for example to cause the second cleaningrobot to forgo cleaning the particular portion of the retail store. Forexample, the second cleaning robot may include no camera, or a cameraincompatible to the desired image capturing.

In some embodiments, the at least one location within the retail storecorresponding to the at least one store shelf for determining the firstadjustment may include a location of the at least one store shelf and/ora location from which camera 1502 may be configured to capture the oneor more images of the at least one portion of the at least one storeshelf.

In some embodiments, the at least one processor may select one of aplurality of alternative cleaning robots 1501 within the retail store(e.g., cleaning robot 1501A, cleaning robot 1501B, etc.) based onplanned routes associated with the plurality of the alternative cleaningrobots. For example, the at least one processor may select a cleaningrobot that is currently closest to the location of the store shelf. Asanother example, the at least one processor may select a cleaning robotthat has the shortest detour from its current route. Alternatively oradditionally, the at least one processor may select a cleaning robotbased on the power status of the plurality of alternative cleaningrobots and/or levels of cleaning provisions of the plurality ofalternative cleaning robots. Alternatively or additionally, the at leastone processor may select a cleaning robot based on one or moreparameters of cameras corresponding to the plurality of alternativecleaning robots. The at least one processor may also cause a firstadjustment to the route of the selected cleaning robot as describedelsewhere in this disclosure.

At step 1713, at least one processor of server 1701 may receive a firstimage acquired by camera 1502 associated with cleaning robot 1501. Thefirst image may include a representation of the at least one portion ofthe at least one store shelf. By way of example, as illustrated in FIG.16, camera 1502 may be configured to capture one or more images of oneor more objects in FOV 1631, which may include a representation of atleast one portion of store shelf 1531. Camera 1502 may transmit(directly or via cleaning robot 1501) one or more captured images toserver 1701.

In some embodiments, camera 1502 may be configured to capture one ormore images according to one or more coverage parameters correspondingto the store shelf and/or one or more product types placed on the storeshelf. One or more coverage parameters may be received from server 1701and/or user device 1702. By way of example, a coverage parametercorresponding to a product type may include at least one of a depthwithin a shelf on which a plurality of products of the product type areplanned to be placed, a percentage of the plurality of products of theproduct type that have been to be captured in an image to enablerecognition, a percentage of the captured portion of one product in animage to enable recognition, a capturing resolution, or a capturingfrequency. In some embodiments, the at least one processor may determinea product type and determine a coverage parameter based on the producttype. For example, the at least one processor may determine a particularproduct type based on a store plan, a store map, a planogram, arealogram, user input, one or more images of the store shelf supportingthe products of the product type, or the like, or a combination thereof.The at least one processor may also determine a coverage parameter basedon the product type by, for example, consulting a lookup table includinga plurality of coverage parameters corresponding to a plurality ofproduct types. Server 1701 may be configured to transmit the coverageparameter to camera 1502.

In some embodiments, the at least one processor may transmit aninstruction to cleaning robot 1501 to clean an area associated with theat least one store shelf.

At step 1714, at least one processor of server 1701 may analyze thefirst image to determine a need for a second image of the at least oneportion of the at least one store shelf. For example, the at least oneprocessor may analyze the first image and determine an occlusion of atleast a part of the at least one store shelf and/or an occlusion of oneor more items placed on the store shelf. By way of example, the at leastone processor may analyze the first image and determine that the atleast one store shelf was blocked by an object (e.g., a person, ashopping cart, etc.) from the view of camera 1502 when the first imagewas captured. In response to the determined occlusion, the at least oneprocessor may determine the need for the second image of the at leastone portion of the at least one store shelf. Alternatively oradditionally, the at least one processor may be configured to analyzethe first image to attempt to recognize a product. The at least oneprocessor may determine the need to capture second image of the at leastone portion of the at least one store shelf in response to a failure torecognize the product. Alternatively or additionally, the at least oneprocessor may analyze the first image and determine a need for an imageof the at least one portion of the at least one store shelf at aresolution higher than a resolution of the first image. Alternatively oradditionally, the at least one processor may analyze the first image anddetermine a need for a second image of the at least one portion of theat least one store shelf captured by the image sensor from a differentposition and/or a different orientation. Alternatively or additionally,the at least one processor may analyze the first image and determine thequantity of items of a product type placed on the store shelf based onthe analysis of the first image. The at least one processor may also beconfigured to determine a need for capturing a second image based on thedetermined quantity. For example, the at least one processor maydetermine the quantity of soda cans placed on the store shelf based onthe analysis of the first image. The at least one processor maydetermine that although there is no need for an immediate restocking ofthe soda, but probably needs restocking after a time period (e.g., anhour) based on the current quantity of soda cans. The at least oneprocessor may also be configured to determine a need for capturing asecond image of the store shelf in an hour to determine whether arestocking is needed. In some embodiments, the at least one processormay determine the time period after which a second image is to be takenbased on the determined quantity of the items of the product type.Alternatively or additionally, the at least one processor may analyzethe first image to calculate at least one convolution of at least partof the first image. The at least one processor may also be configured todetermine a need for a second image based on the calculated at least oneconvolution of at least a portion of the first image. For example, inresponse to a first value of a particular calculated convolution of theat least part of the first image, the at least one processor maydetermine a second image is needed. As another example, in response to asecond value of the particular calculated convolution of the at leastpart of the first image, the at least one processor may determine that asecond image is not needed. By way of example, the first value of theparticular calculated convolution may be indicative of an inadequatesharpness of the first image, and the at least one processor maydetermine that a second image is needed based on the first value of theparticular calculated convolution. The second value of the particularcalculated convolution may be indicative of an adequate sharpness of thefirst image, and the at least one processor may determine that a secondimage is not needed based on the second value of the particularcalculated convolution. In another example, the first value of theparticular calculated convolution may indicate that the first image doesnot include a depiction of a particular item, and the at least oneprocessor may determine that a second image is needed based on the firstvalue of the particular calculated convolution. The second value of theparticular calculated convolution may indicate that the first imageincludes a depiction of the particular item, and the at least oneprocessor may determine that a second image is not needed based on thesecond value of the particular calculated convolution. In someembodiments, a machine learning model may be trained using trainingsamples from previously captured images (by the same camera thatcaptured the image for analysis, or one or more different cameras) todetermine a need for one or more additional images. The at least oneprocessor may use the trained machine learning model to analyze thefirst image and determine a need for a second image of the at least oneportion of the at least one store shelf. A training sample may include asample image and a label indicating whether an additional image isneeded in addition to the sample image.

In some embodiments, the at least one processor may analyze the firstimage to determine an action associated with the at least one portion ofthe at least one store shelf to be performed. The determined action mayinclude at least one of restocking one or more products of a particularproduct type, re-arranging one or more items on the at least one storeshelf, or removing one or more items on the at least one store shelf.The at least one processor may also be configured to transmit to userdevice 1702 an instruction relating to the determined action.Alternatively or additionally, at least one processor may be configuredto transmit to a store management system information related to thedetermined action.

At step 1715, at least one processor of server 1701 may cause a secondadjustment to the route of cleaning robot 1501 within the retail storein response to the determined need. The second adjustment may beconfigured to enable camera 1502 to capture the second image of the atleast one portion of the at least one store shelf. For example, the atleast one processor may determine a new position for cleaning robot 1501from where camera 1502 may be configured to capture one or more secondimages. The at least one processor may also cause a second adjustment tothe route of cleaning robot 1501 such that cleaning robot 1501 may beconfigured to move the new position according to the second adjustment.Camera 1502 may be configured to capture one or more second images atthe new position. Alternatively or additionally, the at least oneprocessor may determine a new orientation of camera 1502 at the oldposition or a new position. For example, the at least one processor maydetermine a new orientation of camera 1502 (for example, tilting a lensof camera 1502 upwards). The at least one processor may also beconfigured to transmit information relating to the new orientation tocamera 1502, which may be configured to capture one or more secondimages with the new orientation.

In some embodiments, the at least one processor may analyze the secondimage to determine an action associated with the at least one portion ofthe at least one store shelf to be performed. The determined action mayinclude at least one of restocking one or more products of a particularproduct type, re-arranging one or more items on the at least one storeshelf, or removing one or more items on the at least one store shelf.The at least one processor may also be configured to transmit to userdevice 1702 an instruction relating to the determined action.Alternatively or additionally, at least one processor may be configuredto transmit to a store management system information related to thedetermined action.

In some embodiments, cleaning robot 1501 may be configured to clean anarea prior to, when, or after camera 1502 captures the first image (orthe second image). For example, cleaning robot 1501 may be configured toclean a first area of the retail store before capturing of the firstimage. Alternatively or additionally, cleaning robot 1501 may beconfigured to clean a second area of the retail store after thecapturing of the second image.

In some embodiments, the first adjustment (and/or the second adjustment)to the route of cleaning robot 1501 may skip (and/or add) one or moreareas to be cleaned by cleaning robot 1501 prior to the first adjustment(and/or the second adjustment). For example, the original route ofcleaning robot 1501 may pass an aisle, and cleaning robot 1501 may beconfigured to clean an area associated with the aisle. The firstadjustment to the route of cleaning robot 1501 may skip that aisle, andcleaning robot 1501 may not clean the area associated with the aisleafter the first adjustment. Server 1701 may select and direct anothercleaning robot to clean the area. For example, server 1701 may determinean adjustment to the route of the selected cleaning robot and cause theselected cleaning robot to clean the area based on the adjustment.Alternatively or additionally, the first adjustment (and/or the secondadjustment) to the route of cleaning robot 1501 may add one or moreparticular areas for cleaning robot 1501 to clean. For example, server1701 may be configured to cause an adjustment to a second cleaning robotthat is to clean one of the one or more particular areas such that thesecond cleaning robot may skip the one of the one or more particularareas. By way of example, the route of the second cleaning robot priorto the adjustment may include a path passing the at least one storeshelf (which may be covered by the route of cleaning robot 1501 afterthe first adjustment). Server 1701 may cause an adjustment to the secondcleaning robot route such that the adjusted route may not include a pathpassing the at least one store shelf.

While the description of process 1710 is provided using server 1701 asan example, one having ordinary skill in art would understand that userdevice 1702 may be configured to perform one or more steps of process1710. the description of process 1710 is provided using one or morecleaning robots as an example, one having ordinary skill in art wouldunderstand that other types of moving devices, such as vehicles,unmanned aerial vehicles, drones, robots, or the like, may also beconfigured to perform one or more functions of a cleaning robotdescribed herein.

In embodiments consistent with the present disclosure, the described“store shelf” or “shelf” may also include one or more pegboards eachproviding multiple peg-hooks for hanging products for display. Forexample, retail store 105 (as shown in FIG. 1) may display products on apegboard that has multiple holes arranged in a matrix with multiple rowsand columns or in another arrangement. A peg-hook may be inserted toeach of the holes, and one or more items may be hung on the peg-hook fordisplay. It is contemplated that the above-described methods and systemsregarding the “store shelf” or “shelf” may also be used to monitor anddetect changes of the items hanging on the pegboards and peg-hooks,wherever applicable.

FIG. 18A is an illustration of an example pegboard 1800, consistent withthe present disclosure. Retail store 105 may have more than one suchpegboard 1800 for displaying products. As shown in FIG. 18A, pegboard1800 has a plurality of peg-hooks 1810 for hanging items 1820. Peg-hooks1810 may be connected to pegboard 1800 in any suitable way. For example,peg-hooks 1810 may be made as an integral part of and thus may not bedetachable from pegboard 1800. As another example, peg-hooks 1810 may bemade as separate modules that may be inserted into holes on pegboard1800 and, when not needed, detached from pegboard 1800. As yet anotherexample, peg-hooks 1810 may be retractable, such that they may beextended or pulled out from pegboard 1800 when they are used for hangingitems 1820, and folded or pushed into pegboard 1800 when they are notused.

The items 1820 hanging on a peg-hook 1810 may be of a single or multipleproduct types. For example, as shown in FIG. 18A, a peg-hook 1810 may beused to hang items 1820A of product type A (e.g., pouches of chips),another peg-hook 1810 may be used to hang items 1820B of product type B(e.g., pouches of crackers), and yet another peg-hook 1810 may be usedto hang items 1820C of product type C (e.g., bags of candies).

Pegboard 1800 and/or peg-hooks 1810 may include sensors 1812 that maysense whether there are items hanging on each peg-hook 1810, and/orsense the change of items hanging on each peg-hook 1810 (e.g., additionor removal of one or more items) and/or may capture data related toitems hanging on each peg-hook 1810 (such as total weight, change inweight, pressure applied by the items, and so forth). Sensors 1812 maybe connected to pegboard 1800 and/or peg-hooks 1810, and configured tosense a parameter (e.g., weight, pressure, force, etc.) reflecting thestatus or change in status of the items hanging on each peg-hook 1810.For example, FIG. 18B shows an exemplary sensor 1812 attached to the topsurface of a peg-hook 1810 or embedded in the upper part of the peg-hook1810, according to some disclosed embodiments. As shown in FIG. 18B,sensor 1812 may be configured to sense a weight of the items hanging onpeg-hook 1810, or sense a pressure applied by the items on peg-hook1810. Increase of the weight and/or pressure may indicate that one ormore items have been added to peg-hook 1810, while decrease of theweight and/or pressure may indicate that one or more items have beenremoved from peg-hook 1810.

As another example, FIG. 18C shows an exemplary sensor 1812 attached tothe bottom surface of a peg-hook 1810 or embedded in the lower part ofthe peg-hook 1810. As shown in FIG. 18C, sensor 1812 may be configuredto sense a pressure applied by peg-hook 1810 to the sensor 1812.Increase of the pressure may indicate that one or more items have beenadded to peg-hook 1810, while decrease of the pressure may indicate thatone or more items have been removed from peg-hook 1810.

As another example, FIG. 18D shows an exemplary sensor 1812 disposedbetween a peg-hook 1810 and a pegboard 1800. As shown in FIG. 18D,sensor 1812 may be a piezoelectric film that may generate a voltagesignal with an amplitude proportional to a pressure applied to the film.Increase of the voltage (i.e., sensed pressure) may indicate that one ormore items have been added to peg-hook 1810, while decrease of thevoltage (i.e., sensed pressure) may indicate that one or more items havebeen removed from peg-hook 1810.

As yet another example, FIG. 18E shows an exemplary sensor 1812 (e.g., aspring) connecting a peg-hook 1810 to a pegboard 1800. As shown in FIG.18D, sensor 1810 may be configured to sense a force applied by peg-hook1810. Increase of the force may indicate that one or more items havebeen added to peg-hook 1810, while decrease of the force may indicatethat one or more items have been removed from peg-hook 1810.

It is contemplated that the examples in FIGS. 18B, 18C, 18D, and 18E arefor illustrative purpose only, and are not the only possible types,configurations, shapes, and locations of sensors 1812 that may be usedin the disclosed embodiments.

As described above, in response to a change in items hanging on apeg-hook 1810, the corresponding sensor 1812 may generate a signalindicating the change. In the disclosed embodiments, peg-hook sensors1812 may transmit the signals to a computer for further processing.According to some embodiments, the computer may be connected to peg-hooksensors 1812 via a wired or wireless network.

For example, each peg-hook sensor 1812 may carry a wirelesscommunication module and may transmit a signal wirelessly to the remotecomputer for further processing. As another example, shown in FIG. 18F,peg-hook sensors 1812 on a pegboard 1800 may be connected and transmitsignals, via communication cables 1814 or through wirelesscommunication, to a communication module 1816 located on the pegboard1800 or in another location. Communication module 1816 may then transmitthe signals to the remote computer via a wired or wireless network (notshown).

According to some embodiments, the computer for processing the sensorsignals may be located on the pegboard 1800 on which peg-hooks 1810 arelocated or in another location. For example, communication module 1816in FIG. 18F may have computing power and act as a processor forprocessing the signals generated by sensors 1812 on pegboard 1800.Communication module 1816 may also send, via a wired or wirelessnetwork, the processing result to a remote computer (e.g., a remoteserver, not shown in FIG. 18F) for further analysis or use.

Example details regarding the computer for processing the signalsgenerated by peg-hook sensors 1812 are described in connection with FIG.19, which is a block diagram representation of an example system formonitoring changes of items hanging on peg-hooks connected to apegboard, consistent with the present disclosure. As shown in FIG. 19,the system may include a pegboard monitoring device 1925, which furthermay include a processing device 1902, a memory interface 1904, a networkinterface 1906, and a peripherals interface 1908. Peripherals interface1908 may be connected, via communication cables or a wireless network,to peg-hook sensors 1812 on a pegboard 1800 (FIG. 18A). In someembodiments, retail store 105 may use a plurality of pegboards 1800, andone or more pegboard monitoring devices 1925 may be used to processsensor signals collected from the plurality of pegboards 1800. Forexample, a pegboard monitoring device 1925 may mounted on each pegboard1800 and analyze sensor signals captured from the pegboard.Alternatively, in some embodiments, one pegboard monitoring device 1925may be used to process the sensor signals collected from two or morepegboards 1800. In another embodiment, two or more pegboard monitoringdevices 1925 may be used to analyze sensor signals captured from asingle pegboard.

Still referring to FIG. 19, processing device 1902, memory interface1904, network interface 1906, and peripherals interface 1908 may beseparate or may be integrated in one or more integrated circuits. Thesecomponents in pegboard monitoring device 1925 may be coupled by one ormore communication buses or signal lines (e.g., bus 1900). It is to beunderstood that pegboard monitoring device 1925 is merely exemplaryimplementation. For example, any operation described in relation topegboard monitoring device 1925 or processing device 1902 may beperformed by one or more other computing devices. In another example,any component stored in memory device 1914 may be stored in one or moreother memory devices.

According to some embodiments, network interface 1906 may be used tofacilitate communication with server(s) 1960 and/or user device(s) 1970and/or a cloud platform and/or other external devices. By way ofexample, server 1960 may be operated by retail store 105 to analyze thedata generated by pegboard monitoring device 1925. User device 1970 maybe a terminal (e.g., smart phone, personal computer, mobile device,augmented reality system, virtual reality system, etc.) used by ashopper or store associate to receive information from pegboardmonitoring device 1925. Network interface 1906 may be an Ethernet portconnected to radio frequency receivers and transmitters and/or opticalreceivers and transmitters. The specific design and implementation ofnetwork interface 1906 may depend on the communications network(s) overwhich pegboard monitoring device 1925 is intended to operate. Forexample, in some embodiments, pegboard monitoring device 1925 mayinclude a network interface 1906 designed to operate over a GSM network,a GPRS network, an EDGE network, a Wi-Fi or WiMax network, a Bluetooth®network, etc.

In the example illustrated in FIG. 19, in addition to being connected topeg-hook sensors 1812, peripherals interface 1908 of pegboard monitoringdevice 1925 may also be connected to at least one image sensor 1910 thatmay be configured to capture image data showing the condition of thepegboard(s) 1800 or items placed on the pegboard(s) associated withpegboard monitoring device 1925. According to some embodiments,peripherals interface 1908 may also be connected to other sensors (notshown), such as a motion sensor, a light sensor, infrared sensor, soundsensor, a proximity sensor, a temperature sensor, a biometric sensor, orother sensing devices to facilitate related functionalities.

Consistent with the present disclosure, pegboard monitoring device 1925may include digital components that collect data from peg-hook sensors1812, and store the data on a memory device 1914 and/or transmit thedata using network interface 1906.

Consistent with the present disclosure, pegboard monitoring device 1925may use memory interface 1904 to access memory device 1914. Memorydevice 1914 may include high-speed, random access memory and/ornon-volatile memory such as one or more magnetic disk storage devices,one or more optical storage devices, and/or flash memory (e.g., NAND,NOR) to store data collected from peg-hook sensors 1812 and/or imagesensor(s) 1910. Memory device 1914 may store operating systeminstructions 1916, such as DARWIN, RTXC, LINUX, iOS, UNIX, LINUX, OS X,WINDOWS, or an embedded operating system such as VXWorkS. Operatingsystem 1916 may include instructions for handling basic system servicesand for performing hardware dependent tasks. In some implementations,operating system 1916 may include a kernel (e.g., UNIX kernel, LINUXkernel, etc.). In addition, memory device 1914 may include sensor fusioninstructions 1918 to facilitate processes and functions related tointegrating and analyzing data collected from peg-hook sensors 1812; andapplication instructions 1920 to perform processes/functions (e.g.,monitoring compliance of product placement on pegboard 1800) in responseto the analysis of the data collected from peg-hook sensors 1812.

The components and arrangements shown in FIG. 19 are examples and arenot intended to limit the disclosed embodiments. As will be appreciatedby a person skilled in the art having the benefit of this disclosure,numerous variations and/or modifications may be made to the depictedconfiguration of pegboard monitoring device 1925. For example, not allcomponents are needed for the operation of pegboard monitoring device1925 in all cases. Any component may be located in any appropriate partof pegboard monitoring device 1925, and the components may be rearrangedinto a variety of configurations while providing the functionality ofthe disclosed embodiments.

In some embodiments, peg-hook sensors 1812 may generate signalsindicating a change of items hanging on the corresponding peg-hooks1810. Specifically, when the items on pegboard 1800 changes (forexample, when a store associate of retail store 105 rearranges orreplenishes the items hanging on pegboard 1800, when a shopper 120 takesitems from or put items back to pegboard 1800, etc.) peg-hook sensors1812 may generate signals indicating these changes. By way of example,processing device 1902 may receive, from a first peg-hook sensor 1812, afirst signal indicating an increase of pressure applied on a firstpeg-hook 1810. In response to receiving the first signal indicating thepressure/weight increase, processing device 1902 may determine that oneor more items have been added to first peg-hook 1810. Processing device1902 may further determine the quantity of the items added to firstpeg-hook 1810, the product type of the newly added items, and/or thetotal quantity of the items hanging on first peg-hook 1810 after theaddition. Processing device 1902 may also perform other actions relatedto the detected addition, as described in more detail below. As anotherexample, processing device 1902 may receive, from a second peg-hooksensor 1812, a second signal indicating a decrease of pressure appliedon a second peg-hook 1810. In response to receiving the second signalindicating the pressure decrease, processing device 1902 may determinethat one or more items have been removed from second peg-hook 1810.Processing device 1902 may further determine the quantity of the itemsremoved from second peg-hook 1810, the product type of the removeditems, and/or the total quantity of the items remaining on secondpeg-hook 1810 after the removal. Processing device 1902 may also performother actions related to the detected removal, as described in moredetail below. In the embodiments consistent with the present disclosure,peg-hook sensors 1812 are not limited to the pressure or weight sensorsdescribed in these examples. For example, peg-hook sensors 1812 may alsobe weight sensors, and processing device 1902 may make similardeterminations and/or actions based on the increase or decrease of thesensed weight change. In some examples, signal indicating a short termchanges (for example, shorter than a selected time period) in thepressure applied on peg-hook 1810 (or in another physical quantitymeasured in relation to peg-hook 1810) may be ignored. In anotherexample, processing device 1902 may interpret signal indicating a shortterm changes as an indication of an interaction of a user with peg-hook1810 or items thereon, and may interpret signal indicating a long termchanges as an indication of a change to the items on peg-hook 1810. Insome examples, signals from one or more peg-hooks 1810 connected topegboard 1800 may be used to preprocess signals from a particularpeg-hook 1810 connected to pegboard 1800, for example to remove noise,to cancel effects of movements external to the particular peg-hook (suchas movement due to interaction with the one or more peg-hooks 1810, dueto interaction with pegboard 1800, etc.) on the particular peg-hook.

According to some embodiments, processing device 1902 may use varioustechniques to determine the product types of the items hanging onpeg-hooks 1810. In one exemplary embodiment, memory device 1914 maystore product recognition models describing the characteristics of eachproduct type (e.g., 1.75 oz. chip pouch, 1.84 oz. candy bar, 1.7 oz.face cream bottle, earphones, etc.) For example, a product recognitionmodel corresponding to a particular product type may include parametersdescribing the product weight, amount of pressure sensed by a peg-hook1810 when an item of the product type is hanging on peg-hook 1810,standard deviation of the weight, standard deviation of the pressure,location of the peg-hooks 1810 designated for hanging the items of theproduct type, image features associated with the product type, etc.Based on the product recognition models and the signals generated bypeg-hook sensors 1812, processing device 1902 may determine the producttypes of the items hanging on peg-hooks 1810. For example, when apeg-hook sensor 1812 senses a change of weight or pressure, processingdevice 1902 may compare the change to the stored product recognitionmodels. Based on the comparison, processing device 1902 may determinethe product type associated with the items added to or removed from thepeg-hook 1810 associated with the peg-hook sensor 1812.

Additionally or alternatively, signals generated by sensors other thanpeg-hook sensors 1812 may also be used to determine the product typesassociated with the items hanging on peg-hooks 1810. For example, asdescribed above, processing device 1902 may also be connected to atleast one image sensor 1910, via peripherals interface 1908 (FIG. 19).Image sensor 1910 may be mounted (for example, to store shelves, walls,ceilings, floors, refrigerators, checkout stations, displays,dispensers, rods which may be connected to other objects in the retailstore, and so forth) and configured to take images of the items hangingon peg-hooks 1810. Processing device 1902 may extract features from theimage data generated by image sensor 1910, and recognize the producttypes of the items hanging on peg-hooks 1810, by comparing the extractedfeatures to a product image model. In some cases, processing device 1902may also perform a sensor fusion technique to compare the productrecognition results based on the peg-hook sensors 1812 and imagesensor(s) 1910, and determine the product types based on the comparison.For example, processing device 1902 may use the image data tocross-check the product type results determined based on the peg-hooksensor data, and assign confidence values to the determined producttypes.

In one exemplary embodiment, different peg-hooks 1810 may bepre-assigned to and reserved for hanging different product types. Forexample, a particular pegboard 1800 may be designated by retail store105 for hanging cosmetics products. As another example, peg-hooks 1810in a particular area of a pegboard 1800 may be reserved for hanging aparticular brand of chips. In this embodiment, memory device 1914 maystore a predetermined peg-hook map indicating which peg-hooks 1810 areused for hanging what product types (i.e., corresponding relationshipbetween peg-hooks 1810 and the product types). Based on the peg-hookmap, processing device 1902 may determine the product type of the itemshanging on each peg-hook.

According to some embodiments, processing device 1902 may cause variousactions based on the signals generated by peg-hook sensors 1812. hi oneexemplary embodiment, in response to a signal indicating a change ofitems hanging on a peg-hook 1810, processing device 1902 may determine aquantity of the items remaining in the peg-hook 1810. For example, if apeg-hook sensor 1812 generates a signal indicating a pressure increase,processing device 1902 may compare the amount of pressure increase to aproduct recognition model associated with the product type hanging onpeg-hook sensor 1812, and determine the number of items added topeg-hook 1810 or the total number of items remaining on peg-hook 1810after the addition. If the product recognition model indicates a unitpressure corresponding to one item of this product type, processingdevice 1902 may determine the number of items newly added to peg-hook1810 by dividing the amount of pressure increase with the unit pressure,and/or determine the total number of items on peg-hook 1810 after theaddition by dividing the total pressure sensed by peg-hook sensor 1812with the unit pressure. As another example, if a peg-hook sensor 1812generates a signal indicating a weight decrease, processing device 1902may compare the amount of weight decrease to a product recognition modelassociated with the product type hanging on peg-hook sensor 1812, anddetermine the number of items removed from peg-hook 1810 or the numberof items remaining on peg-hook 1810 after the removal. hi particular, ifthe product recognition model indicates a unit weight corresponding toone item of this product type, processing device 1902 may determine thenumber of items removed from peg-hook 1810 by dividing the amount ofweight decrease with the unit weight, and/or determine the number ofitems remaining on peg-hook 1810 after the removal by dividing the totalweight sensed by peg-hook sensor 1812 with the unit weight. In someexamples, a convolution of a signal generated by peg-hook sensors 1812may be calculated, in response to a first value of the calculatedsignal, a first action may be caused, and in response to a second valueof the calculated signal, a second action may be caused, the secondaction may differ from a first action.

According to some embodiments, based on the product type associated withthe items added to or removed from a peg-hook 1810, processing device1902 may cause different actions. In one exemplary embodiment, when thesignal generated by a peg-hook sensor 1812 indicates that one or moreitems are added to the peg-hook 1810 associated with the peg-hook sensor1812, processing device 1902 may determine whether the product type ofthe newly added items is the same as the product type associated withthe peg-hook 1810. For example, processing device 1902 may determine theproduct type associated with the peg-hook 1810 based on the peg-hook map(i.e., the corresponding relationship between peg-hooks 1810 and theproduct types) stored in memory device 1914, or based on a comparison ofthe product recognition models with the peg-hook condition (e.g.,pressure or weight) sensed by peg-hook sensor 1812 before the items areadded. If it is determined that the product type of the newly addeditems matches the product type associated with the peg-hook 1810,processing device 1902 may determine the number of the items hanging onpeg-hook 1810 after the addition, and/or transmit the determined number,via communications network 1950, to server 1960 and/or user device 1970for further processing. If, however, it is determined that the producttype of the newly added items does not match the product type associatedwith the peg-hook 1810, processing device 1902 may transmit anotification, via communications network 1950, to server 1960 and/oruser device 1970. The notification may alert a user (e.g., a storeassociate of retail store 105) about the mismatch of product typesand/or prompt the user to reorganize the items to ensure that items ofthe correct product type are hanging on a peg-hook 1810.

In one exemplary embodiment, when the signal generated by a peg-hooksensor 1812 indicates that one or more items have been removed from apeg-hook 1810 associated with the peg-hook sensor 1812, processingdevice 1902 may determine the product type of the removed items. Forexample, processing device 1902 may determine the product type of theremoved items based on the peg-hook map (showing the correspondingrelationship between peg-hooks 1810 and the product types), or based ona comparison of the change sensed by peg-hook sensor 1812 (e.g.,pressure or weight decrease) with the product recognition models.Processing device 1902 may further determine the quantity of the producttype remaining on a peg-hook 1810 or pegboard 1800 after the removal. Ifit is determined that the remaining quantity is below a presetthreshold, processing device 1902 may transmit a notification, viacommunications network 1950, to server 1960 and/or user device 1970. Thenotification may prompt a user (e.g., a store associate of retail store105) to restock the determined product type to peg-hook 1810 or pegboard1800. Additionally or alternatively, the notification may prompt theuser to rearrange the items hanging on pegboard 1800. For example, thedetected removal of items may suggest the associated product type ispopular among customers. Accordingly, the notification may prompt theuser to hang the determined product type at more prominent locations onpegboard 1800 (e.g., hanging the product type on peg-hooks 1810 close tothe average height of the customers).

According to some embodiments, processing device 1902 may cause anaction based on the conditions of multiple peg-hook sensors 1812. In oneexemplary embodiment, processing device 1902 may receive, from a firstpeg-hook sensor 1812, a signal indicating that one or more items areremoved from a first peg-hook 1810, and receive, from a second peg-hooksensor 1812, a signal indicating that one or more items are added to asecond peg-hook 1810. If it is determined that the removed items andadded items are the same product type, processing device 1902 maygenerate an alert prompting a user (e.g., an employee of retail store105) to move some items from the second peg-hook 1810 to the firstpeg-hook 1810.

According to some embodiments, based on the signals generated bypeg-hook sensors 1812, processing device 1902 may generate adistribution map of the items hanging on a pegboard 1800. In oneexemplary embodiment, the distribution map may include informationshowing the quantities and product types of the items hanging onpeg-hooks 1810. FIG. 18G shows an exemplary distribution map of theitems hanging on pegboard #20 in retail store 105. Processing device1902 may generate the distribution map based on the product type anditem quantity determined for each peg-hook 1810. Processing device 1902may further cause the distribution map to be displayed on a displaydevice, e.g., a display device connected to pegboard monitoring device1925 via peripherals interface 1908, or server 1960 or user device 1970connected to pegboard monitoring device 1925 via communications network1950. As shown in FIG. 18G, the distribution map may use various symbolsor shapes to represent different product types, and highlight thepeg-hooks 1810 that need a user's attention. For example, thedistribution map in FIG. 18G uses dash lines to show that peg-hook1810-12 (i.e., peg-hook 1810 at row 1, column 2) is empty, which alertsa user to restock peg-hook 1810-12. The distribution map also useshatched symbols to indicate that the quantity of the items hanging onpeg-hook 1810-25 is low, and alerts the user to restock peg-hook1810-25.

Still referring to FIG. 18G, processing device 1902 may also determine,based on the distribution map, whether the items hanging on pegboard #20comply with a planogram. Processing device 1902 may highlight violationsof the planogram on the distribution map. For example, as shown in FIG.18G, processing device 1902 may determine that peg-hook 1810-23 violatesthe planogram by having two different product types hanging on it, andhighlight the wrong product type (shown in solid black triangle) thatneeds to be removed from peg-hook 1810-23. As another example,processing device 1902 may determine that the items hanging on peg-hook1810-34 are a wrong product type according to the planogram, andhighlight the items hanging on peg-hook 1810-34 to indicate that theyneed to be moved to a different peg-hook 1810. As described above,processing device 1902 may cause the distribution map to be displayed ona display device (e.g., server 1960 or user device 1970), such that auser (e.g., an employee of retail store 105) may view whether peg-hook#20 complies with the planogram.

According to some embodiments, in response to the signal generated bypeg-hook sensors 1812, processing device 1902 may cause other types ofsensors to be activated to capture conditions of pegboard 1800. Forexample, referring to FIG. 19, when it is determined that the producttype or quantity of the items added to or removed from a peg-hook 1810cannot be determined based on the output of peg-hook sensors 1812,processing device 1902 may activate an image sensor 1910 associated withthe peg-hook 1810 (e.g., a camera facing an area including the peg-hook1810) and cause image sensor 1910 to capture one or more images showinga scene including the peg-hook 1810.

Processing device 1902 may analyze the one or more captured images, forexample the processing device may extract features from the capturedimages, compare the extracted features to the product recognitionmodels, and may determine the product type or quantity based on thecomparison. Processing device 1902 may further determine whether theproduct type or quantity complies with the planogram (e.g., whether theitem added to the peg-hook 1810 have the correct product type, orwhether the quantity of the items remaining on the peg-hook 1810 is toolow). If a planogram violation is detected, processing device 1902 maycause a captured image to be displayed on a display device (e.g., server1960 or user device 1970), such that a user (e.g., an employee of retailstore 105) may view the items hanging on pegboard 1800 and takeappropriate actions.

According to some embodiments, processing device 1902 may determinewhether a shopper qualifies for frictionless checkout. For example, inresponse to a change detected by peg-hook sensors 1812, processingdevice 1902 may activate image sensors 1910 to capture one or moreimages showing the customers who are shopping the items hanging onpegboard 1800. If the identities of the customers, as well as theproduct types and quantities of the items taken by the customers frompegboard 1800 may be determined based on the data generated by peg-hooksensors 1812 and image sensors 1910, processing device 1902 maydetermine that the customers qualify for frictionless checkout. However,if the customer identities, product types, or item quantities cannot bedetermined, of if no bank account linked to the customers may be found,processing device 1902 may determine that the customers do not qualifyfor frictionless checkout, and may provide an alert to the customers orretail store 105. For example, processing device 1902 may transmit analert to a user device 1970 associated with a customer, indicating thatthe customer is not eligible for frictionless checkout. Additionally oralternatively, processing device 1902 may transmit the alert to a server1960 operated by retail store 105. Server 1960 may determine the numberof customers not qualified for frictionless checkout at a given time,and deploy sufficient resources to handle the checkout process (e.g.,activate more checkout terminals, or call for more store employee tohelp with the checkout line).

According to some embodiments, when a person or a robot in retail store105 hangs new product types on pegboard 1800 that do not have associatedproduct recognition models, processing device 1902 may execute variouslearning programs 1922 stored in memory device 1914 (FIG. 19) to developthe product recognition models for the new product types. Specifically,processing device 1902 may determine that the output of a peg-hooksensor 1812 (e.g., a pressure or weight change) does not match anyexisting product recognition models. In response, processing device 1902may determine that the items added to the peg-hook 1810 associated withthe peg-hook sensor 1812 has a new product type. Processing device 1902may further execute learning programs 1922 to extract features from theoutput of the peg-hook sensor 1812, as well as the output of other typesof sensor (e.g., image data showing the condition of the associatedpeg-hook 1810), and train a new product recognition model using theextracted features. In one exemplary embodiment, the new productrecognition module may be a mathematical function that assigns a weightto each of the extracted features. During the training process, theweights may be determined based on a training data set which includestraining features and training product types associated with thetraining features. In some embodiments, learning programs 1922 mayinclude a convolutional neural network (CNN) with a plurality ofconvolutional layers for iteratively extracting the features andtraining the new product recognition model,

According to some embodiments, processing device 1902 may use the outputof peg-hook sensors 1812 to determine a shopping pattern and assistretail store 105 in optimizing the planogram for pegboard 1800. Forexample, the shopping pattern may indicate the locations of “popular”peg-hooks 1810 on which the hanging items are more often taken by theshoppers. The shopping pattern may also indicate the popularity ofdifferent product types according to how fast the product types are soldout. Processing device 1902 may generate, based on the shopping pattern,a decision model for arranging the product types on pegboard 1800. Forexample, the decision model may prompt retail store 105 to hanghigh-value product types on the “popular” peg-hooks 1810. As anotherexample, the decision model may suggest to retail store 105 that morepeg-hooks 1810 should be used to hang popular product types, or theproduct types on promotion should be moved to more prominent locationson pegboard 1800.

Consistent with the disclosed embodiments, as shown in FIG. 19,processing device 1902 may perform the above-described operations byexecuting the instructions and programs stored in the memory device1914. For example, sensor fusion instructions 1918, when executed byprocessing device 1902, may cause processing device 1902 to compare thedata generated by different peg-hook sensors 1812 and other types ofsensors (e.g., image sensors 1910) and determine the conditionassociated with pegboard 1800 and/or peg-hooks 1810. As another example,application instructions 1920, when executed by processing device 1902,may cause processing device 1902 to cause various actions (e.g.,generating the distribution map, generating a notification or alert,etc.) based on the condition associated with pegboard 1800 and/orpeg-hooks 1810. As another example, learning programs 1922, whenexecuted by processing device 1902, may cause processing device 1902 totrain new product recognition models based on the output of peg-hooksensors 1812 and other types of sensors (e.g., image sensors 1910).

FIG. 20 provides a flowchart of an exemplary method 2000 for detectingchanges of items hanging on peg-hooks connected to a pegboard,consistent with the present disclosure. In one exemplary embodiment,memory device 1914 may store one or more computer programs correspondingto method 2000. When executed by processing device 1902, the one or morecomputer programs may cause processing device 1902 to perform some orall of the operations in method 2000. As shown in FIG. 20, method mayinclude the following steps 2002-2008.

At step 2002, method 2000 includes receiving, from a first peg-hook 1810connected to a pegboard 1800, a first indication indicative of a changeof items hanging on first peg-hook 1810. For example, processing device1902 may receive the first indication from first peg-hook 1810. Thefirst indication may include an indication of a weight change caused bythe items hanging on first peg-hook 1810. As another example, the firstindication may include an indication of a pressure changed caused by theitems hanging on first peg-hook 1810.

At step 2004, method 2000 includes receiving, from a second peg-hook1810 connected to pegboard 1800, a second indication indicative of achange of items hanging on second peg-hook 1810. For example, processingdevice 1902 may receive the second indication from second peg-hook 1810.Similarly, the second indication may include an indication of a weightor pressure change caused by the items hanging on second peg-hook 1810.

At step 2006, method 2000 includes, in response to the first indication,causing an action related to a first product type. For example, based onthe first indication, processing device 1902 may determine that one ormore items are added to or removed from first peg-hook 1810. Processingdevice 1902 may determine the product type of the added or removeditems. Processing device 1902 may also determine the quantity of theadded or removed items. Processing device 1902 may further cause one ormore other actions based on the determined quantity and/or product type.For example, if it is determined that the quantity of the items hangingon first peg-hook 1810 is low, processing device 1902 may generate anotification indicating a need to restock first peg-hook 1810. Asanother example, if it is determined that the items added to firstpeg-hook 1810 are not the product type designated for first peg-hook1810, processing device 1902 may provide a warning regarding themismatch and prompt a user to move the items to other peg-hooks 1810that are associated with the correct product type.

At step 2008, method 2000 includes, in response to the secondindication, causing an action related to a second product type. Forexample, similar to step 2006, processing device 1902 may determine theproduct type and/or quantity of the items added to or removed fromsecond peg-hook 1810.

Based on the determined product type and/or quantity, processing device1902 may further cause other actions in a manner similar to thatdescribed in step 2006.

According to method 2000, actions may also be caused based on ananalysis of both the first and second indications. For example,processing device 1902 may use the first and second indications togenerate a distribution map (e.g., FIG. 18G) showing the product typesand quantities of the items hanging on each peg-hook 1810 connected topegboard 1800. Based on the distribution map, processing device 1902 maydetermine whether the items hanging on pegboard 1800 complies with aplanogram, and generate a warning if a possible planogram violation isdetected. Based on the distribution map, processing device 1902 may alsogenerate a notification for rearranging items hanging on pegboard 1800(e.g., notification indicating a need to move items from over crowedpeg-hooks 1810 to empty peg-hooks 1810).

As described above, cameras may be installed off shelf to capturepictures depicting the products displayed on a retail shelf. Inaddition, various types of on-shelf sensors may be used to detect thecondition of the products on the shelf. These sensors may generatenon-picture or non-visual sensor data which, combined with the visualdata produced by the off-shelf cameras, may be used to identify thepresence (or absence) of a product and/or determine the product typeassociated with the product. As used herein, “identifying a product”refers to the operations for determining the condition of one or moreproducts on a retail shelf, such as detecting the presence of a product,determining the product type associated with the product, determiningthe quality (e.g., freshness) of the product, determining the quantityof the products with the same product type, determining the arrangementof the products on the retail shelf, etc.

Specifically, as described in connection with FIGS. 4A-4C and 6A-6C,various cameras may be used to capture visual data about a retail shelf.The cameras may be installed on an adjacent shelf, a ceiling, a wall, ahand-held device used by a store associate, a customer or a visitor ofthe store, a store patrolling robot, etc. Moreover, as described inconnection with FIGS. 8A and 8B, the retail shelf may include aplurality of on-shelf sensors (e.g., detection elements 801A and 801Bshown in FIG. 8A and detection elements 851A and 851B shown in FIG. 8B)for detecting conditions of the products on the retail shelf. It iscontemplated that the above description about the cameras and on-shelfsensors also applies to the present embodiment and is incorporatedherein.

FIG. 21 is a schematic diagram illustrating a sensor fusion techniquefor identifying the products on a retail shelf, consistent with thepresent disclosure. As shown in FIG. 21, on-shelf sensor data 2100 andimage data 2150 may be input to one or more product recognition models2160. The output of product recognition models 2160 may include producttype information 2190 regarding the products displayed on the retailshelf, and other information related to at least part of the productsdisplayed on the retail shelf (such as the condition of the products,the quality (e.g., freshness) of the products, the quantity of theproducts, the arrangement of the products, etc.). In some examples,on-shelf sensor data 2100 may include data captured using sensorsconfigured to be positioned between a shelf and products placed on theshelf.

In one exemplary embodiment, on-shelf sensor data 2100 may include datagenerated by one or more weight sensors 2130. The sensor data (e.g., oneor more weight signals) generated by weight sensors 2130 may indicateweights that match profiles of particular products (e.g., certain brandof coffee mugs or a particular type of pre-packaged spinach). The weightsignals may also be representative of actual weight values associatedwith a particular product type or, alternatively, may be associated witha relative weight value sufficient to identify the product and/or toidentify the presence of a product.

In one exemplary embodiment, on-shelf sensor data 2100 may include datagenerated by one or more pressure sensors 2132. The sensor data (e.g.,one or more pressure signals) generated by pressure sensors 2132 mayindicate a pressure caused by a product on the retail shelf. Thepressure signals may indicate an average pressure caused by the product.The average pressure is represented by a single numerical value.Alternatively or additionally, the pressure signals may indicate apressure distribution showing the shelf surface areas where pressure issensed (e.g., the contact surface between the product and the shelf). Inone example, the pressure distribution may further show the magnitude ofthe pressure at each point or at selected points of the contact surface.

In one exemplary embodiment, on-shelf sensor data 2100 may include datagenerated by one or more footprint sensors 2134. The sensor datagenerated by footprint sensors 2134 may indicate the impression madeand/or left by a product on a shelf surface. For example, footprintsensors 2134 may include one or more touch sensors that may sense thetouch points or touch surfaces at which a product contact a retailshelf. As another example, footprint sensors 2134 may include one ormore of the above-described pressure sensors 2132 for detecting apressure distribution caused by the product. The pressure distributionmay not only show where the product contacts the shelf surface (i.e.,where non-zero pressure is detected), but also the magnitude of thepressure at each point of the contact surface. Items of differentproduct types may differ on their centers of mass, structures, shapes,states (solid, liquid, or gas), etc., and thus may cause differentpressure distributions on the shelf. As such, the pressure distributionsmay represent “footprints” that are characteristic of the differentproduct types.

In one exemplary embodiment, on-shelf sensor data 2100 may include datagenerated by one or more light sensors 2136 capable of detecting ambientlight. The sensor data (e.g., one or more light information signals)generated by light sensors 2136 may be indicative of ambient light beingblocked by particular products, which correspond to shadows caused bythe products. The light signals may also be representative of actuallight patterns associated with a particular product type or,alternatively, may be associated with light patterns sufficient toidentify the product and/or to identify the presence of a product.

In one exemplary embodiment, on-shelf sensor data may include datagenerated by one or more acoustic sensors 2138. The sensor datagenerated by acoustic sensors 2138 may include sound signals that matchprofiles of particular products. For example, the sound signals mayindicate the vibrations of the products that are characteristic of thematerial composition in different product types.

It is contemplated that the disclosed on-shelf sensor data 2100 is notlimited to above-described types of sensor data. For example, on-shelfsensor data 2100 may also include data generated by one or more motionsensors, proximity sensors, capacitive sensors, resistive sensors,inductive sensors, infrared sensors, ultrasonic sensors, temperaturesensors, etc.

FIG. 22 is a block diagram representation of an example system foridentifying products on a retail shelf based on on-shelf sensor data andcamera-generated visual data, consistent with the present disclosure. Asshown in FIG. 22, the system may include a product type analyzer 2225,which further may include a processing device 2202, a memory interface2204, a network interface 2206, and a peripherals interface 2208.Peripherals interface 2208 may be connected, via communication cables ora wireless communication, to various on-shelf sensors, such as one ormore weight sensors 2230, pressure sensors 2232, footprint sensors 2234,light sensors 2236, acoustic sensors 2238, etc. The on-shelf sensors maygenerate non-picture or non-visual on-shelf sensor data 2100 indicatingstatus and/or condition of the products displayed on a store shelf.Peripherals interface 2208 may also be connected, via communicationcables or a wireless communication, to one or more image sensors 2245that generate image data 2150 depicting the condition of the productsdisplayed on the retail shelf.

Processing device 2202, memory interface 2204, network interface 2206,and peripherals interface 2208 may be separate or may be integrated inone or more integrated circuits. These components in product typeanalyzer 2225 may be coupled by one or more communication buses orsignal lines (e.g., bus 2200).

According to some embodiments, network interface 2206 may be used tofacilitate communication with server(s) 2260 and/or user device(s) 2270.By way of example, server 2260 may be operated by retail store 105 toanalyze the data generated by product type analyzer 2225. User device2270 may be a terminal (e.g., smart phone, smart watch, tablet, personalcomputer, mobile device, wearable device, virtual reality system,augmented reality system, etc.) used by a shopper or store employee toreceive information from product type analyzer 2225. Network interface2206 may be a port (e.g., an Ethernet port) connected to radio frequencyreceivers and transmitters and/or optical receivers and transmitters.The specific design and implementation of network interface 2206 maydepend on the communications network(s) over which product type analyzer2225 is intended to operate. For example, in some embodiments, producttype analyzer 2225 may include a network interface 2206 designed tooperate over a GSM network, a GPRS network, an EDGE network, a Wi-Fi orWiMax network, a Bluetooth® network, etc.

Consistent with the present disclosure, product type analyzer 2225 mayinclude digital components that collect data from on-shelf sensors(e.g., weight sensors 2230, pressure sensors 2232, footprint sensors2234, light sensors 2236, acoustic sensors 2238, etc.) and image sensors2245, and store the data on a memory device 2214 and/or transmit thedata using network interface 2206. Additionally or alternatively,product type analyzer 2225 may preprocess the data collected from theon-shelf sensors and image sensors to obtain preprocessed data, and maystore the preprocessed data on a memory device 2214 and/or transmit thepreprocessed data using network interface 2206. Additionally oralternatively, product type analyzer 2225 may analyze the data collectedfrom the on-shelf sensors and image sensors to an analysis result, andmay store the analysis result on a memory device 2214 and/or transmitthe analysis result using network interface 2206. Additionally oralternatively, product type analyzer 2225 may analyze the data collectedfrom the on-shelf sensors and image sensors to an analysis result, andmay cause an action based on the analysis result. For example, inresponse to a first analysis result, product type analyzer 2225 maycause a particular action, and in response to a second analysis result,product type analyzer 2225 may forgo or withhold causing the particularaction. In another example, in response to a first analysis result,product type analyzer 2225 may cause a first action, and in response toa second analysis result, product type analyzer 2225 may cause a secondaction, the second action may differ from the first action. Somenon-limiting examples of such actions may include capturing ofadditional data (such as additional image data, additional data from theon-shelf sensors, additional data from other sensors, etc.), providingnotifications, transmitting information, storing information, turning adevice on or off, and so forth.

Consistent with the present disclosure, product type analyzer 2225 mayuse memory interface 2204 to access memory device 2214. Memory device2214 may include high-speed, random access memory and/or non-volatilememory such as one or more magnetic disk storage devices, one or moreoptical storage devices, and/or flash memory (e.g., NAND, NOR) to storedata collected from on-shelf sensors (e.g., weight sensors 2230,pressure sensors 2232, footprint sensors 2234, light sensors 2236,acoustic sensors 2238, etc.) and image sensors 2245. Memory device 2214may store operating system instructions 2216, such as DARWIN, RTXC,LINUX, iOS, UNIX, LINUX, OS X, WINDOWS, or an embedded operating systemsuch as VXWorkS. Operating system 2216 may include instructions forhandling basic system services and for performing hardware dependenttasks. In some implementations, operating system 2216 may include akernel (e.g., UNIX kernel, LINUX kernel, etc.). In addition, memorydevice 2214 may include sensor fusion programs 2218 to facilitateprocesses and functions related to integrating and analyzing datacollected from on-shelf sensors (e.g., weight sensors 2230, pressuresensors 2232, footprint sensors 2234, light sensors 2236, acousticsensors 2238, etc.) and image sensors 2245. Memory device 2214 may alsoinclude application instructions 2220 to perform processes/functions(e.g., monitoring compliance of product placement on a retail shelf ormonitoring the inventory level of certain product types) in response tothe analysis of the data collected from the on-shelf sensors and imagesensors. Memory device 2214 may also include learning programs 2222,described below.

The components and arrangements shown in FIG. 22 are examples and arenot intended to limit the disclosed embodiments. As will be appreciatedby a person skilled in the art having the benefit of this disclosure,numerous variations and/or modifications may be made to the depictedconfiguration of product type analyzer 2225. For example, not allcomponents are needed for the operation of product type analyzer 2225 inall cases. Any component may be located in any appropriate part ofproduct type analyzer 2225, and the components may be rearranged into avariety of configurations while providing the functionality of thedisclosed embodiments.

Still referring to FIG. 22, processing device 2202 may use the datagenerated by the on-shelf sensors and image sensors to identify theproducts displayed on a retail shelf. In particular, processing device2202 may receive, via bus 2200 and peripherals interface 2208, datacaptured by the on-shelf sensors, such as weight sensors 2230, pressuresensors 2232, footprint sensors 2234, light sensors 2236, acousticsensors 2238, etc. The on-shelf sensors may be positioned between atleast part of a retail shelf and one or more products placed on the atleast part of the retail shelf. Processing device 2202 may also receive,via bus 2200 and peripherals interface 2208, one or more images capturedby image sensors 2245. The images may include a representation of atleast part of the retail shelf and at least one of the one or moreproducts. Processing device 2202 may then analyze the data (i.e.,on-shelf sensor data 2100 shown in FIG. 21) captured by the on-shelfsensors and the images (i.e., image data 2150 shown in FIG. 21) capturedby image sensors 2245 to determine the presence (or absence) of one ormore products on the retail shelf, the product type associated with theone or more products, and other information related to the one or moreproducts.

Consistent with the disclosed embodiments, on-shelf sensor data 2100 andimage data 2150 do not have to cover the same part of the retail shelf.For example, the image sensors' line of sight may be blocked by certainobstructions (e.g., a product may be occluded by its surroundingproducts, or the products placed on a lower shelf may be blocked byupper shelves). In contrast, the on-shelf sensors may capture dataregarding the products that are not capable of being captured by theimage sensors. As described below in more detail, by combining theon-shelf sensory data with the images, processing device 2202 may inferthe condition (e.g., product type, quantity, quality, etc.) of theproducts not shown in the images, by using one or more productrecognition models 2160.

According to some embodiments, processing device 2202 may use variousways to determine the product types of the products displayed on aretail shelf. In one exemplary embodiment, memory device 2214 may storeproduct recognition models 2160 describing the characteristics of eachproduct type (e.g., 1.75 oz. chip pouch, 1.84 oz. candy bar, 1.7 oz.face cream bottle, earphones, etc.). For example, a product recognitionmodel 2160 corresponding to a particular product type may includeparameters describing the product weight, amount of pressure caused byan item of the product type on a retail shelf, footprint of an item ofthe product type, ambient light pattern associated with an item of theproduct type, acoustic pattern associated with an item of the producttype, image features associated with the product type, etc. Based onproduct recognition models 2160 and the signals generated by theon-shelf sensor and/or image sensors, processing device 2202 maydetermine the product types of the products displayed on a retail shelf.

For example, in one embodiment, product recognition models 2160 mayinclude the unit weights associated with a plurality of product typesrespectively. The unit weight is the weight of a single item (e.g., abox of tissues, a pouch of snacks, a bottle of wine, etc.). The datagenerated by weight sensors 2130 may indicate the weights of one or moreproducts placed on a retail shelf. Processing device 2202 may comparethe sensed weights to the unit weights in product recognition models2160, to determine one or more matched product types.

In one exemplary embodiment, product recognition models 2160 may includethe average pressures associated with a plurality of product typesrespectively, or pressure distributions that are characteristic of theplurality of product types respectively. The data generated by pressuresensors 2132 may indicate the average pressures or pressuredistributions caused by one or more products placed on a retail shelf.Processing device 2202 may compare the sensed average pressures orpressure distributions to the pressure information in productrecognition models 2160, to determine one or more matched product types.

In one exemplary embodiment, product recognition models 2160 may includethe typical footprints associated with a plurality of product typesrespectively. The data generated by footprint sensors 2134 may indicatethe footprints of one or more products placed on a retail shelf.Processing device 2202 may compare the sensed footprints to thefootprint information in product recognition models 2160, to determineone or more matched product types.

In one exemplary embodiment, product recognition models 2160 may includethe characteristic ambient light patterns (or shadows) associated with aplurality of product types respectively. The data generated by lightsensors 2136 may indicate the ambient light patterns or shadow patternsof one or more products placed on a retail shelf. Processing device 2202may compare the sensed ambient light or shadows to the ambient light orshadow information in product recognition models 2160, to determine oneor more matched product types.

In one exemplary embodiment, product recognition models 2160 may includethe typical acoustic wave information associated with a plurality ofproduct types respectively. The data generated by acoustic sensors 2138may indicate the changes caused to an acoustic wave by one or moreproducts placed on a retail shelf. Processing device 2202 may comparethe sensed changes to the acoustic wave to the acoustic information inproduct recognition models 2160, to determine one or more matchedproduct types.

In one exemplary embodiment, product recognition models 2160 may includecharacteristic visual features (e.g., shape, color, brightness, etc.)associated with a plurality of product types respectively. The imagesgenerated by image sensors 2245 may include depth information, colorinformation, brightness information, etc., that can be extracted todetermine the shape, color, brightness, etc., of the objects in theimages. Processing device 2202 may compare the extracted image featuresto the corresponding image features in product recognition models 2160,to determine one or more matched product types.

The above-described on-shelf sensor data 2100 and image data 2150 arefor illustrative purpose only, and are not the only sensor data or imagedata that can be used in the disclosed embodiments. A single parameteror feature (e.g., weight, pressure, footprint, ambient light, sound,image, etc.) extracted from on-shelf sensor data 2100 and image data2150 may only be able to provide a crude description of the producttype, but not enough to distinguish between different product types. Forexample, a box of cereal and a box of cake powder may have the sameweight; thus, weight alone may not be sufficient to distinguish them.Similarly, a can of tomato sauce and a bottle of olive oil may both havea cylindrical shape or leave a round-shaped footprint on the retailshelf, and thus the shape or footprint alone may not be enough todistinguish them. However, by integrating the multiple parameters andfeatures extracted from on-shelf sensor data 2100 and image data 2150,processing device 2202 may determine the product type accurately. Forexample, using a combination of two or more of weight, pressure,footprint, ambient light, shape, sound, color, and brightness,processing device 2202 may accurately distinguish the product typeassociated with a product.

Processing device 2202 may execute one or more sensor fusion programs2218 to compare and analyze on-shelf sensor data 2100 and image data2150. Specifically, in one exemplary embodiment, processing device 2202may analyze on-shelf sensor data 2100 to determine a plurality ofalternative candidate product types (e.g., multiple candidate producttypes having the same unit weight) that could be present on a retailshelf. Processing device 2202 may then analyze image data 2150 depictingthe retail shelf to select a correct product type from the plurality ofcandidate product types. For example, processing device 2202 maydetermine, based on image data 2150, the shape and color of the productson the retail shelf, and select, based on product recognition models2160, a candidate product type that matches the determined shape andcolor.

In one exemplary embodiment, processing device 2202 may analyze imagedata 2150 of a retail shelf to determine a plurality of alternativecandidate product types that could be present on the retail shelf.Processing device 2202 may further analyze on-shelf sensor 2100 toselect a correct product type from the plurality of candidate producttypes. For example, based on image data 2150, processing device 2202 maydetermine that the items displayed on the retail shelf are coffee mugs,but may not be able to further distinguish the logos or surface textureson different brands of coffee mugs. However, by analyzing the footprintsof the coffee mugs, processing device 2202 may determine that they are aparticular coffee mugs that have a square (rather than round or oval)bottom. In another example, based on image data 2150, processing device2202 may determine that the items displayed on the retail shelf are of aspecific kind, but be unable to determine (or unable to determine insufficient confidence) a size and/or a weight and/or a volume of theitems (for example, determine that the items are bottles of a particulartype of beverage but unable to determine a size, a weight or a volume ofa bottle). However, by analyzing the weight of and/or footprints ofand/or pressure caused by the items, processing device 2202 maydetermine that the size and/or weight and/or volume of the items.

In one exemplary embodiment, processing device 2202 may analyze imagedata 2150 of a retail shelf and determine that the products on the shelfbelong to a first product type. Moreover, processing device 2202 mayanalyze on-shelf sensor data 2100 associated with the retail shelf todetermine that the items on the shelf belong to a second product type.Processing device 2202 may then compare the first and second producttypes. If the first product type (determined based on image data 2150)is the same as the second product type (determined based on on-shelfsensor data 2100), processing device 2202 may assign a high confidencevalue to the determined product type. If the first product type differsfrom the second product type, processing device 2200 may analyzeadditional information (i.e., information other than on-shelf sensordata 2100 and image data 2150) to determine the product type associatedwith the products. The additional information may include, but is notlimited to, historic information regarding the retail shelf (e.g., theproduct types previously displayed on the retail shelf), place of theretail shelf in a retail store (e.g., whether the retail shelf islocated in an aisle for condiments or in a section of the store for homecleaning products), planogram information, product types of nearbyproducts, shelf labels, height of the retail shelf, 3D images, depthimages, stereo images, scans of a visual code (such as bar codes),images of higher quality or that are captured using a differentcapturing settings, and so forth. In one example, processing device 2202may query a person (such as a store associate, a customer, a visitor,etc.) or a robot for the additional information. Based on the analysisof the additional information, processing device 2202 may determinewhich of the first and second product types is the correct one.Alternatively, processing device 2202 may determine, based on theanalysis of the additional information, that the products on the shelfbelong to a third product type that differs from both the first andsecond product types. For example, the first product type (determinedbased on image data 2150) may be a salad bowl, while the second producttype (determined based on on-shelf sensor data 2100) may be a cookingpot. However, if the additional information indicates that the retailshelf is located in a children's toy section and a height of the retailshelf makes it reachable by children, processing device 2202 maydetermine that the correct product type is a children's beach sandbucket.

In one exemplary embodiment, product recognition models 2160 may beimplemented using a convolutional network (CNN). In performing thedisclosed sensor fusion technique, processing device 2202 may executethe CNN to extract features from on-shelf sensor data 2100 and imagedata 2150. Each product recognition model 2160 may be a mathematicalfunction that assigns a weight to each of the extracted features. Duringthe inference of product recognition models 2160, processing device 2202may iteratively refine the weights associated with the extractedfeatures. The output of product recognition models 2160 may include, forexample, a plurality of possible product types and their probabilitysores. Processing device 2202 may determine a product type based on theoutput, e.g., selecting the product type with the highest probabilityscore.

In one exemplary embodiment, processing device 2202 may extract thefeatures of an image by calculating at least one convolution of at leastpart of the image. For example, processing device 2202 may apply a firstkernel to image data 2150 to extract features associated with theimages. Moreover, processing device 2202 may convert on-shelf sensordata 2100 to an array of values, and extract features of on-shelf sensordata 2100 by calculating at least one convolution of at least part ofthe array of values. For example, processing device 2202 may converton-shelf sensor data 2100 into a plurality of variables describing thedetected weight, pressure, footprint, ambient light, sound, etc., andthen organize the variables in one or more data arrays. Processingdevice 2202 may apply a second kernel to the one or more data arrays toextract features associated with on-shelf sensor data 2100. In someexamples, processing device 2202 may determine the product type based onthe calculated at least one convolution of the at least part of theimage and the on-shelf sensor data 2100, may determine the product typebased on the calculated at least one convolution of at least part of thearray of values and the image, may determine the product type based onthe calculated at least one convolution of the at least part of theimage and the calculated at least one convolution of at least part ofthe array of values and the image, and so forth.

In one exemplary embodiment, processing device 2202 may further executelearning programs 2222 to train product recognition models 2160. Duringthe training process, the parameters of product recognition models 2160may be determined based on a training data set which includes a trainingimage of at least one training product on a training shelf, on-shelfsensor data associated with the at least one training product and/or thetraining shelf, and a product type associated with the at least onetraining product. In some embodiments, learning programs 2222 mayinclude a CNN with a plurality of convolutional layers for iterativelyextracting the features and training product recognition models 2160. Insome embodiments, learning programs 2222 may include other machinelearning algorithms, as described above.

According to some disclosed embodiments, in addition to determining theproduct types, processing device 2202 may also use on-shelf sensor data2100 and/or image data 2150 to determine other conditions associatedwith the products displayed on a retail shelf, such as the quantity,quality, or facing directions of the displayed products. hi oneexemplary embodiment, processing device 2202 may determine the quantityof the products displayed on a retail shelf by determining the number ofproducts shown in an image of the shelf. If, however, the image does notcover the entirety of the shelf, processing device 2202 may relyon-shelf sensor data 2100 (e.g., pressure data, weight data, footprintdata, etc.) to determining the number of products in the part of theshelf not shown in the image. This way, by combining image data 2150 andon-shelf sensor data 2100, processing device 2202 may accuratelydetermine the total number of products displayed on the retail shelf.

In one exemplary embodiment, processing device 2202 may determine thequality of the products on a retail shelf by using on-shelf sensor data2100 and/or image data 2150. For example, if an image of a retail shelfshows the spinach thereon has become yellowish and the acoustic sensordata also indicates the water content in the spinach has decreased tocertain level, processing device 2202 may determine that the spinach isno longer fresh. As another example, if image data 2150 shows a milkcontainer has changed its shape or the light sensor data shows that theambient light pattern associated with the milk has changed, processingdevice 2202 may determine that the milk has gone bad.

In one exemplary embodiment, processing device 2202 may determine thefacing directions of the products on a retail shelf by using on-shelfsensor data 2100 and/or image data 2150. For example, the image data mayindicate that a product on a shelf has a different orientation from therest of the products on the shelf, and the footprint sensor data mayalso indicate that the product has its front side facing to an inwarddirection of the shelf. In this case, processing device 2202 maydetermine that the product is displayed with a wrong facing direction.

According to some embodiments, processing device 2202 may executeapplication instructions 2220 to cause various actions based on on-shelfsensor data 2100 and/or image data 2150. Specifically, the execution ofapplication instructions 2222 may cause processing device 2202 todetermine an action associated with the retail shelf. Some non-limitingexamples of such actions may include restocking of the retail shelf,removing products from the retail shelf, rearranging products on theretail shelf, replacing products on the retail shelf, ordering productsassociated with the retail shelf, etc. The execution of applicationinstructions 2220 may also cause processing device 2202 to generate andprovide information configured to cause the performance of the actions.For example, processing device 2202 may transmit a report regarding theproduct condition on a shelf to server 2260 and/or user device 2270 viacommunications network 2250. Server 2260 may run various programs tomonitor and manage the store inventory. After receiving the report fromprocessing device 2202, server 2260 may automatically place an order forproduct types that have a low inventory level. As another example, userdevice 2270 may be a hand-held device (e.g., smart phone, tabletcomputer, laptop computer, etc.) used by a store employee. Based on thereport from processing device 2202, user device 2270 may display amessage prompting the store employee to perform various actions, such asrestocking the retail shelf, removing products from the retail shelf,rearranging products on the retail shelf, replacing products on theretail shelf, etc.

According to some embodiments, to improve the system efficiency andreduce operation cost, image data 2150 regarding a retail shelf may becaptured and processed when it is needed. Specifically, processingdevice 2202 may analyze on-shelf sensor data 2100 to determine whetherthere is a need to capture image data 2150 regarding the retail shelf.In response to the determination that it is needed to capture image data2150, processing device 2202 may trigger the capture of image data 2150.For example, when it is determined that on-shelf sensor data 2100 aloneis not enough to determine the product type or detect a productcondition on the retail shelf, processing device 2202 may cause an imagesensor mounted to another retail shelf to capture the image, cause adome camera to move and capture the image, cause a mobile robot tonavigate to a location corresponding to the retail shelf and capture theimage, cause a store associate to use a hand-held camera to capture theimage, and so forth.

Consistent with the disclosed embodiments, as shown in FIG. 22,processing device 2202 may perform the above-described operations byexecuting the instructions and programs stored in the memory device2214. For example, sensor fusion programs 2218, when executed byprocessing device 2202, may cause processing device 2202 to compareon-shelf sensor data 2100 with image data 2150, and determine thecondition of the products displayed on a retail shelf. As anotherexample, application instructions 2220, when executed by processingdevice 2202, may cause processing device 2202 to cause various actions(e.g., generating a notification regarding the need for restocking aretail shelf, generating a report regarding the condition of theproducts displayed on the retail shelf, etc.) based on on-shelf sensordata 2100 and/or image data 2150. As another example, learning programs2222, when executed by processing device 2202, may cause processingdevice 2202 to train product recognition models 2160 based on trainingon-shelf sensor data and training image data.

FIG. 23 provides a flowchart of an exemplary method 2300 for identifyingproducts on a store shelf, consistent with the present disclosure. Inone exemplary embodiment, memory device 2214 may store one or morecomputer programs corresponding to method 2300. When executed by atleast one processor (e.g., processing device 2202), the one or morecomputer programs may cause the at least one processor to perform someor all of the operations in method 2300. As shown in FIG. 23, method mayinclude the following steps 2302-2306.

At step 2302, method 2300 includes receiving on-shelf sensor data 2100captured using a plurality of sensors positioned between at least partof a retail shelf and one or more products placed on the at least partof the retail shelf. For example, on-shelf sensor data 2100 may include,but are not limited to, data captured by one or more weight sensors2130, pressure sensors 2132, footprint sensors 2134, light sensors 2136,acoustic sensors 2138, etc. for example, processing device 2202 may becommunicatively connected to these sensors and receive on-shelf sensordata 2100 from these sensors via peripherals interface 2208.

At step 2304, method 2300 includes receiving image data 2150 of the atleast part of the retail shelf and at least one of the one or moreproducts. For example, image data 2150 may be generated by one or moreimage sensors 2145, which may include an image sensor mounted to anotherretail shelf, a dome camera above the retail shelf, an image sensor of amobile robot, a hand-held camera, and so forth. Processing device 2202may be communicatively connected to image sensors 2145 and receive imagedata 2150 from image sensors 2145 via peripherals interface 2208.

At step 2306, method 2300 includes analyzing the captured on-shelfsensor data 2100 and image data 2150 to determine a product type of theone or more products. For example, processing device 2202 may run asensor fusion program 2218 to extract features from on-shelf sensor data2100 and image data 2150, and analyze the extracted features using oneor more product recognition models 2160. Based on the output of the oneor more product recognition models 2160, processing device 2202 maydetermine a product type of the one or more products. In some examples,a machine learning model may be trained using training examples todetermine information related to products placed on shelf from imagedata and captured on-shelf sensor data, and step 2306 may use thetrained machine learning model to analyze the captured on-shelf sensordata 2100 (e.g., the data received by step 2302) and image data 2150(e.g., the image received by step 2304) to determine a product type ofthe one or more products and/or other information related to the one ormore products. An example of such training example may include a sampleimage and a sample on-shelf sensor data corresponding to a sample shelf,together with a label indicative of information related to productsplaced on at least part of the sample shelf (such as product type,condition, quantity, quality, arrangement, facings, and so forth).

According to method 2300, in addition or alternatively to the producttype information, other aspects of the product condition on the retailshelf may also be determined based on on-shelf sensor data 2100 and/orimage data 2150. For example, processing device 2202 may determine thequantity, quality, and/or facing directions of the products displayed onthe retail shelf. Moreover, besides on-shelf sensor data 2100 and imagedata 2150, additional information may also be analyzed to cross-check orsupplement the analysis result of on-shelf sensor data 2100 and imagedata 2150. For example, the additional information may include historicinformation regarding the products displayed on the retail shelf, placeof the retail shelf in a retail store, planogram information, producttypes of nearby products, shelf labels, height of the retail shelf, andso forth. Further, actions may also be caused based on the determinedproduct type and other aspects of the product condition. For example,processing device 2202 may cause the restocking of the retail shelf,removing products from the retail shelf, rearranging products on theretail shelf, replacing products on the retail shelf, ordering productsassociated with the retail shelf, and so forth.

The data generated by method 2300 and/or product type analyzer 2225 maybe used to facilitate further analysis. For example, time series of thegenerated data (e.g., product type data, quantity data, quality data,condition data, arrangement data, facings data, planogram compliancedata, etc.) may be aggregated or constructed, and the time series may beanalyzed to provide additional information or to select actions. In oneexample, the generated data and/or the time series data may be used tovalidate information arriving from other sources. For example, a systemfacilitating frictionless shopping may provide an indication of one ormore products picked by a shopper, and the generated data and/or thetime series data may be used to validate the indicated one or moreproducts.

In some situations, a person may use an application, such as on asmartphone, to navigate a walkable environment, such as a retail store.In these situations, when a user is far away from a destination in thestore, it can be advantageous to provide the user with a map-like viewinterface to help the person with self-orientation and navigation withinthe store. As the person approaches the destination in the store, amap-like view interface may become less helpful, and it may be moreadvantageous to provide the user with an augmented reality viewinterface to help the person with self-orientation and navigation in asmaller space, such as an aisle, to identify, for example, a particularshelf. The embodiments described below, operable with other embodimentsdisclosed herein, discuss these interfaces and how they may beimplemented to provide meaningful navigational assistance to a person.

FIG. 24 depicts an exemplary navigation assistance map-view userinterface 2400. In some embodiments, capturing device 125, output device145, or any other device may display user interface 2400. For example,user interface 2400 may be displayed on a touchscreen, and a user mayinteract with graphical elements in user interface 2400 by touching oneor more of the graphical elements.

User interface 2400 may include a first function selection area 2402,which may include at least one button or other graphical element, whichmay be selectable by a user. For example, first function selection area2402 may include one or more buttons that, when selected, may: present adifferent user interface on a display, refresh a map, zoom into aportion of the map, zoom out of a portion of the map, or cause anotherchange to the visual elements displayed (e.g., transition to a productselection interface, transition to navigation assistanceaugmented-reality-view user interface 2500, etc.)

User interface 2400 may also include a map 2404, which may be a map of aretail store or other environment (e.g., a scaled visual depiction of anarea within a retail store). Map 2404 may also include one or more shelfindicators 2406, which may indicate and/or correspond to physicalshelving units or other product display structures within a retailstore. In some embodiments, map 2404 may include information associatedwith obstacles (e.g., display stands), walkable areas, non-walkableareas, employee-only areas, and so forth, which may also have associatedindicators to assist a user in navigating a retail store or otherenvironment. For example, map 2404 may include information associatedwith a size and/or position of an obstacle that may block a line ofsight (e.g., between a user and a target destination). In someembodiments, map 2404 may also include visual indicators for objects notnecessarily associated with a product, such as a support pillar for theretail store, a checkout counter, and the like. In some embodiments, map2404 may include a user location indicator 2408, which may indicate orcorrespond to a location of a user or user device (e.g., a mobiledevice) within a retail store associated with map 2404. For example, auser device may determine its location within a retail store by usingelectromagnetic signals, such as GPS signals, Wi-Fi signals, or signalsfrom an indoor localization system. In some embodiments, map 2404 mayalso include a route indicator 2410, which may indicate a route (e.g., awalking route) through a portion of the retail store. For example, routeindicator 2410 may include one or more lines that visually connect userlocation indicator 2408 to a destination indicator 2412. In someembodiments, route indicator 2410 may include distance information, suchas text that denotes a distance between two points along a user's route(e.g., 5 meters, 50 meters, etc.). A destination indicator 2412 mayindicate or correspond to a user's destination, which may be particularshelf and/or product location within the retail store. In someembodiments, a device displaying user interface 2400 may determine aproduct location corresponding to a product identified by a user atanother user interface, and may place destination indicator 2412 at aplace on map 2404 corresponding to the real-world location of theidentified product. In some embodiments, destination indicator 2412 mayinclude information related to a product or a shelf. For example,destination indicator 2412 may include a product name, a productidentifier, a shelf height, a relative shelf placement (e.g., “secondshelf from the floor-level shelf”), or the like. In some embodiments,map 2404 may be an augmented reality map including one or more images orother visual imagery of an in-store environment to provide to a user foraugmented reality navigation. An augmented reality map may be separatefrom, or integrated with, an overhead view map (e.g., as shown in FIG.24). For example, map 2404 may include location data (e.g., coordinatedata) linked to a map location and one or more images.

User interface 2400 may include a second function selection area 2414,which may include at least one button or other graphical element, whichmay be selectable by a user. For example, second function selection area2414 may include one or more buttons that, when selected, may present auser interface including device options, center map 2404 around a user'slocation, or center map 2404 around a product location. Other types ofdevice functions are possible and the preceding are exemplary. In someexamples, first function selection area 2402 and/or second functionselection area 2414 may be presented as an overlay over map 2404.

FIG. 25 depicts an exemplary navigation assistanceaugmented-reality-view user interface 2500. In some embodiments,capturing device 125, output device 145, or any other device may displayuser interface 2500. For example, user interface 2500 may be displayedon a touchscreen, and a user may interact with graphical elements inuser interface 2500 by touching one or more graphical elements. In someembodiments, user interface 2500 may display image data captured by acamera or other imaging device (e.g., a camera connected to a devicedisplaying user interface 2500).

User interface 2500 may include an augmented reality display area 2502,which may include an image (e.g., from a video stream) of a user'senvironment and/or visual overlays. For example, a device, such ascapturing device 125 or output device 145, may capture one or moreimages using a camera, and may integrate at least portions of the one ormore images into augmented reality display area 2502, such as bydisplaying an image and placing one or more visual overlays on theimage. In some embodiments, augmented reality display area 2502 mayinclude a number of shelves 2504 or other objects in an environment of auser. For example, shelves 2504 may be shelves within a retailenvironment, and may correspond to shelf indicators 2406.

As mentioned above, in some embodiments, user interface 2500 may includeone or more visual overlays placed upon an image of a user'senvironment. For example, augmented reality display area 2502 mayinclude a destination overlay indicator 2506, which may be a symbol,icon, outline, color, image distortion, or other visual indicatorassociated with a user's destination. In some embodiments, destinationoverlay indicator 2506 may correspond to a shelf and/or product. Forexample, destination overlay indicator 2506 may correspond to a productselected by a user at a user interface other than user interface 2500,to a product from a shopping list corresponding to the user, to aproduct associated with a coupon corresponding to the user, to an area(e.g., a shelf) and/or a product corresponding to a task assigned to theuser, and so forth. In some embodiments, destination overlay indicator2506 may be overlaid within augmented reality display area 2502 at alocation that corresponds to a location of destination indicator 2412.In some embodiments, destination overlay indicator 2506 may be anoutline in a shape of a selected product. In some examples, the image inaugmented reality display area 2502 may be a display of a live feedcaptured using an image sensor (for example, an image sensor included ina device displaying user interface 2500), for example together with adisplay of one or move overlays, such as destination overlay indicator2506 and/or route overlay indicator 2508. In one example, when the imagein augmented reality display area 2502 changes (for example, due tomovement of a device displaying user interface 2500, due to movement ofa camera capturing the image, etc.), the position of the overlays maychange according to the changes in the image. For example, the locationof destination overlay indicator 2506 may change to a location adjacentto the location of a selected item (such as a product, a shelf, a label,etc.) in the image, the location of route overlay indicator 2508 maychange to a location of the route, and so forth. Additionally oralternatively, destination overlay indicator 2506 may pulse, flash,increase in transparency, decrease in transparency, or otherwise changein visual appearance, which may occur in response to a user's locationbeing within one or more threshold distances of a destination location(e.g., of a product). For example, the destination indicator 2506 maypulse with increasing frequency as a user's location approaches thedestination location. User interface 2500 may also include other visualoverlays, such as route overlay indicator 2508, which may be a line,arrow, grouping of markers, or other visual indicator of a route for auser (e.g., a walking route within a retail store). In some embodiments,route overlay indicator 2508 may be overlaid within augmented realitydisplay area 2502 across an area corresponding to route indicator 2410.Route overlay indicator 2508 may be overlaid between and/or uponobjects, such as shelves 2504. In some embodiments, destination overlayindicator 2506 and/or route overlay indicator 2508 may have a partialdegree of transparency, which may allow a user to view a portion of anenvironment covered by a visual overlay, while still being able tounderstand information conveyed by the visual overlay. Any number ofdestination overlay indicators 2506 and/or route overlay indicators2508, as well as other types of overlay indicators, may be placed withinaugmented reality display area 2502.

User interface 2500 may also include an augmented-reality-view settingselection area 2510, which may include at least one button or othergraphical element, which may be selectable by a user. For example,augmented-reality-view setting selection area 2510 may include one ormore buttons that, when selected, may: adjust a transparency of at leastone overlay indicator, adjust contrast of an image or indicator, adjustbrightness of an image or indicator, alter a color scheme of augmentedreality display area 2502, toggle a visual alert option, toggle anauditory alert option, toggle a haptic alert option, or otherwise adjusthow information is shown within user interface 2500. In some examples,augmented-reality-view setting selection area 2510 may be presented asan overlay over augmented reality display area 2502.

Additionally or alternatively to user interface 2400, map 2404 may bepresented using an augmented reality system (such as augmented realityglasses), for example with a high opacity parameter. Additionally oralternatively to user interface 2500, destination overlay indicator 2506and/or route overlay indicator 2508 may be presented using an augmentedreality system (such as augmented reality glasses). For example,destination overlay indicator 2506 may be presented in a locationadjacent to or over the location of a selected item (such as a product,a shelf, a label, etc.), route overlay indicator 2508 may be presentedin a location indicative of the route, and so forth. In some examples,when a head of a user wearing augmented reality glasses moves (or otherpart of a user associated with angling an augmented reality device), thelocation of the destination overlay indicator 2506 and/or route overlayindicator 2508 moves to maintain the relative location of the overlaysto selected items in the environment of the user, for example tomaintain the location of destination overlay indicator 2506 in alocation adjacent to or over the location of the selected item, to keepthe location of route overlay indicator 2508 in the location indicativeof the route. In some examples, when the head of the user wearing theaugmented reality glasses moves, the location of map 2404 in thedisplay(s) within the augmented reality glasses may stay the same (orsubstantially the same, for example within a selected number of pixelsof the original locations, where the selected number of pixels may beless than 2, less than 5, less than 10, less than 50, less than 100,more than 100, and so forth).

FIG. 26 depicts a flowchart of exemplary process 2600 for providingvisual navigation assistance in retail stores. For purposes ofillustration, in the following description, reference is made to certaincomponents of system 100. For example, any combination of steps ofprocess 2600 may be performed by at least one processor of a device sucha handheld device (e.g., a smartphone, a tablet, a mobile station, apersonal digital assistant, a laptop, and more), a wearable device(e.g., smart glasses, a smartwatch, a clip-on camera), and/or server.Examples of such devices (e.g., capturing device 125, server 135) aredescribed above. It will be appreciated, however, that otherimplementations are possible and that other components may be utilizedto implement the exemplary process 2600. It will also be readilyappreciated that the illustrated method can be altered to modify theorder of steps, repeat steps, delete steps, or further includeadditional steps. While certain aspects are described in the context ofa retail store, it is appreciated that any or all of the steps describedcould be implemented in other environments, such as a warehouse,fulfillment center, stockpile, or other place where a user may attemptto navigate to locate an item.

At step 2602, at least one processor may receive a first indoorlocation, which may be a first indoor location of a user within a retailstore. For example, the first indoor location may be read from a memory,may be received from an external device, may be determined based onanalysis of data (such as an analysis of image data determined using animage sensor to determine a location of the image sensor), may bedetermined using an indoor positioning system, and so forth. In someembodiments, the first indoor location may be determined using positiondata of a device (e.g., a device having at least one processorperforming process 2600), which may include GPS location data and/orindoor positioning data. In some embodiments, such as where positiondata, image data, and the like are exchanged between devices, process2600 may include establishing a wireless connection with a mobile deviceof the user (e.g., a wireless connection between a device and imageprocessing unit 130, a wireless connection between a device and asatellite, etc.).

At step 2604, at least one processor may receive a target destination,which may be a target destination within a retail store. For example,the target destination may be read from a memory, may be received froman external device, may be determined based on analysis of data (forexample, based on an analysis of a shopping list, based on an analysisof a task, etc.). In some embodiments, a target destination may beassociated with a product selected at a mobile device of the user (e.g.,selected from a shopping list, selected from a list of search results,etc.). By way of example and without limitation, a user may select a canof peas (or other item) at the mobile device, and the mobile device maydetermine a location of the can of peas within a retail store anddesignate that location as the target destination. In some embodiments,the mobile device may determine a location of a product by retrievingthe location from a data structure associating product identifiers withproduct locations, which may be maintained in a number of places, suchas at database 140 or locally at the mobile device.

At step 2606, at least one processor may provide first navigationinformation, which may be first navigation data provided to a userthrough a first visual interface. In some embodiments, the first visualinterface may include at least one of an aisle identifier, a retail areaidentifier (e.g., “produce”, “household items”, etc.), a shelfidentifier, or a product identifier. For example, the second visualinterface may include aspects of navigation assistance map-view userinterface 2400. Additionally or alternatively, the first visualinterface may include an image of a product, which may be a product thata user selected at a user interface of a mobile device.

At step 2608, at least one processor may receive a second indoorlocation, which may be a second indoor location of a user within aretail store. For example, the second indoor location may be read from amemory, may be received from an external device, may be determined basedon analysis of data (such as an analysis of image data captured using animage sensor to determine a location of the image sensor), may bedetermined using an indoor positioning system, and so forth. In someembodiments, the second indoor location may be determined using positiondata of a device implementing part or all of process 2600, which mayinclude GPS location data and/or indoor positioning data. In someembodiments, the at least one processor may receive the second indoorlocation after providing the first navigation data. In some embodiments,at least one of the first or second indoor locations may be based onposition data determined by a position sensor of a mobile device of theuser.

For example, the mobile device may have an accelerometer, a geomagneticfield sensor, a camera, an infrared sensor, or other sensor fordetecting environmental context of the mobile device to determine itsposition in an environment. In some embodiments, the position data mayinclude position data determined at least in part based on a strength ofa signal received at the mobile device from a network device placed inthe retail store. In one example, a signal may be a radio wave, Wi-Fisignal, or other wireless electromagnetic wave. In some embodiments, theposition data may include position data determined at least in partbased on a time difference between two signals received at the mobiledevice from two network devices (e.g., wireless access points, routers,etc.) placed in the retail store. Additionally or alternatively, theposition data may include position data determined based on apositioning system that uses multiple signals to triangulate a positionof a device (e.g., a Wi-Fi positioning system).

At step 2610, at least one processor may determine whether the secondindoor location is within a selected area around the target destination.For example, at least one processor may determine whether the secondindoor location is within a selected area that is within a thresholddistance from the target destination, within a selected area having aparticular radius from the target destination, etc. A selected area maybe selected by a user and/or determined according to a program orapplication (e.g., a program or application implementing process 2600).For example, in some embodiments, a user may select an area of 25 metersfrom a target destination (e.g., at a user interface). Any appropriatedistance may be selected. In some embodiments, the selected area may bebased on a radius from a target destination (e.g., within a 20-meterradius of the target destination). In other embodiments, the selectedarea may be based on a navigable distance between a user and a targetdestination (e.g., within a 20-meter navigable distance from the targetdestination). In some embodiments, if the at least one processordetermines that the second indoor location is not within a selected areaaround the target destination, it may provide, or continue to provide,first navigation data (e.g., at step 2606). If the at least oneprocessor determines that the second indoor location is within aselected area around the target destination, the at least one processormay proceed to step 2612. In some embodiments, process 2600 may alsoinclude activating a camera sensor of a mobile device of the user, andthe activating may be in response to the determination that the secondindoor location is within a selected area around the target destination.

In some embodiments, the selected area around the target destination maybe selected based on an aisle including the target destination. Forexample, the selected area may include one or more distances across awalkable area through at least part of an aisle between the targetdestination and a location of a user. In another example, the selectedarea may include an area in which there is a direct line of sight to thetarget destination and/or to an object corresponding to the targetdestination (such as a product, a shelf, a label, etc.). For example,the at least one processor may analyze map data (e.g., of map 2404) todetermine at least one area having a direct line of sight to the targetdestination (e.g., based on the location of the user). In someembodiments, the selected area around the target destination may beselected based on at least one of a store shelf associated with thetarget destination or a product associated with the target destination.For example, a user may designate a product and/or store shelf at amobile device, which the mobile device may designate as the targetdestination. Based on the location of the target destination, the mobiledevice may determine the selected area as an area centered around thetarget destination (e.g., a 30-meter radius around the targetdestination) or as a maximum walkable distance from the targetdestination (e.g., an area including all walkable distances from thetarget destination equal to or less than 30 meters). In someembodiments, the selected area may not include the first indoorlocation.

At step 2612, the at least one processor may provide second navigationinformation, which may be second navigation data provided to a userthrough a second visual interface (e.g., displayable at a mobile device,at an augmented reality system, etc.). In some embodiments, the at leastone processor may provide second navigation data in response to adetermination that the second indoor location is within the selectedarea around the target destination. In some embodiments, the firstvisual interface and/or the second visual interface may be provided to auser via a mobile device, such as a mobile phone. In some embodiments,the second visual interface may include a visual indication of at leastone of a product, a store shelf, an aisle, the target destination, adirection to follow, or a route to follow. For example, the secondvisual interface may include aspects of navigation assistanceaugmented-reality-view user interface 2500.

In some embodiments, the second visual interface may differ from thefirst visual interface. For example, the second visual interface may bean augmented reality visual interface, which may provide localnavigation assistance information to a user (e.g., navigation assistanceaugmented-reality-view user interface 2500) and the first visualinterface may be a map view of the retail store, which may provide amap-like view of an area to a user (e.g., navigation assistance map-viewuser interface 2400). In some embodiments, process 2600 may includereceiving, from a mobile device of the user, at least one image of aphysical environment of the user. The physical environment maycorrespond to a retail store, such that the image may include an aisle,a shelving unit, a product, or any other store structure or item.

In some embodiments, process 2600 may include receiving an imagecaptured by a camera of a mobile device of the user. In someembodiments, process 2600 may also include calculating at least oneconvolution of the image. For example, process 2600 may includerepresenting the image as a matrix of values or a tensor of values(e.g., values corresponding to pixel information, such as color,saturation, hue, brightness, value, etc.) and applying a kernel to thematrix or the tensor to calculate a convolution (which may be used toperform edge detection in the image, sharpen portions of the image,etc.). In some embodiments, process 2600 may also include using thecalculated at least one convolution to determine at least one of thefirst or second indoor locations. For example, image processing unit 130may compare a convolution of an image (or images) or a function of theconvolution of the image to convolutions of another image (or otherimages) or to a threshold, which may be associated with known locations,and which may be stored at database 140. Image processing unit 130 maydetermine that the convolution of the captured image matches (e.g.,pixel values being within a threshold amount) a convolution of an imageassociated with a known location, and determine that the known locationcorresponds to the first or second indoor location. In some embodiments,the convolutions or images associated with known locations or thethresholds may have been generated as part of a visual mapping process(e.g., a process to generate all or part of a map 2404). of an area(e.g., retail store). In one example, in response to a first value of acalculated convolution of at least part of the image, a first locationmay be determined, and in response to a second value of the calculatedconvolution of the at least part of the image, a second location may bedetermined. In some embodiments, the second location may differ from thefirst location.

In some embodiments, at least a portion of the second visual interfacemay be generated based on data captured by a camera of a mobile deviceof a user (e.g., by a mobile phone, by capturing device 125, etc.). Inone example, the captured data may be visual data. In some embodiments,the second visual interface may include a presentation of at least partof an image (e.g., visual data) captured by the camera of the mobiledevice of a user.

Process 2600 may also include determining at least one location in theat least one image, such as a location of a product, target destination,or other location for presenting to a user. In some embodiments,determining the location in the at least one image may include analyzinga distance, analyzing a color, analyzing a color gradient, shapedetection, edge detection, or other image analysis technique (e.g.,using convolutions, as discussed above). In some embodiments,determining at least one location in the at least one image may involvedetermining one or more distances in the at least one image based onlocation data, position data, edge detection, etc. For example, a mobiledevice may access map information describing distances or dimensions ofopen spaces and/or objects in an environment (e.g., a retail store) andmay access position data indicating a current position of the mobiledevice and/or a current orientation of the mobile device. A mobiledevice may identify, such as through edge detection, an end of ashelving unit in an image, or, based on a current position and/ororientation of the mobile device, a point along the shelving unitbetween the mobile device and the end of the shelving unit where atarget destination (e.g., a product) is present. The mobile device maythen cause a visual indicator to display at that point within the image(e.g., within an augmented reality image).

In some embodiments, the at least one processor may determine at leastone location in the at least one image based on at least one of: alocation of the mobile device, an orientation of the mobile device, or aproduct selected at the mobile device. In some embodiments, at least onelocation in an image may also be determined based on information from atleast one sensor, such as an image sensor, weight sensor, pressuresensor, light sensor, or a device networked with a mobile deviceperforming process 2600 (e.g., using a signal strength between awireless access point and a mobile device). Process 2600 may alsoinclude placing a visual indicator on the determined at least onelocation in the at least one image. For example, a mobile device maydetermine that its location (e.g., a user's location) is near a firstend of an aisle, that a selected product location is near a second endof the aisle, and that the camera of the mobile device is oriented toface the second end of the aisle. Then, the mobile device may determinean image location in the image corresponding to a real-world location ofthe selected product, and may place a visual indicator (e.g.,destination overlay indicator 2506) at that image location.

As discussed above, a retail store may place products on store shelvesaccording to a planogram, which may indicate how and where specificretail products should be placed on shelves or displays. Occasionally, aretail store or other entity may modify or alter these planograms, whichmay trigger a need to rearrange products on store shelves. For example,contractual obligations may change, seasonal products may be introduced,a layout of a store may change, or various other events affectingplacement of products may occur. In other examples, products on storeshelves that are not placed according to a planogram (for example, dueto negligence, due to a lack of planogram) may need to be rearranged tocomply with an existing or a new planogram. Accordingly, storeassociates of the retail store or other workers may need to reposition,add, remove, and/or replace products on the shelves.

While relatively minor modifications or changes to a planogram aregenerally manageable, a complete remodel of a planogram can be asignificant undertaking. In particular, removing and replacing all (or asubstantial percentage) of the products on a retail shelf may takeseveral hours to complete and such a large task may deter or discouragethe workforce responsible for carrying out the change. A completeremodeling may also render the shelving unit out of service for a longperiod of time, thereby negatively affecting sales and customerexperience at the retail store. Large remodels may also have a cascadingeffect on other display spaces within the store if products in theremodeled planogram must be moved to or from other spaces. Moreover,adhering to an entirely new planogram may be challenging for retailstore associates, which may negatively affect planogram complianceand/or sales figures for the store.

In view of these and other challenges, techniques for effectively andefficiently managing planogram remodeling are needed. In particular,where a target planogram is known in advance, the disclosed systems andmethods may break a large planogram remodeling task into smallmanageable tasks. In some embodiments, this may include using regularstore operations, such as restocking operations, to complete at leastsome of these small tasks, thereby reducing the overall cost and levelof effort. The disclosed embodiments may further allow a retailworkforce to become gradually accustomed to parts of the new planogramsand may avoid long out-of-service periods. The disclosed embodimentstherefore provide, among other advantages, improved efficiency,convenience, and functionality over prior art planogram compliance andinventory management systems.

As described in detail throughout the present disclosure, the disclosedembodiments may include receiving images of at least part of a shelf. Asnoted above, a shelf may refer to any structure used for displayingproducts in a retail environment. FIG. 27A illustrates an example image2700 including at least part of a shelf 2702, consistent with thepresent disclosure. Image 2700 may be acquired by an image sensorincluded in the retail store, such as image sensor 310 included incapturing device 125. The capturing device may take various forms or maybe mounted in various locations, as described throughout the presentdisclosure. In some embodiments, the capturing device may be mountedadjacent to the shelf on an additional shelf. For example, theadditional shelf may be placed above the shelf and the image sensor maybe mounted below the additional shelf. As another example, theadditional shelf may be placed across an aisle from the shelf.Accordingly, image 2700 may be captured using an image sensor such asimage capturing devices 125A, 125B, and/or 125C. In some embodiments,image 2700 may be captured by a handheld device, such as a personalmobile device. For example, the handheld device may be carried by astore associate of a retail store in which the shelf is included (e.g.,device 125D), a customer of a retail store in which the shelf isincluded, a visitor to a retail store in which the shelf is included, aparticipant in a crowdsourcing platform, a secret shopper, or any otherindividuals that may capture images in a retail store. As anotherexample, image 2700 may be captured by a robotic device, such as a roboton a track (e.g., capturing device 125E), a drone (e.g., capturingdevice 125F), and/or a robot that may move on the floor of the retailstore (e.g., capturing device 125G).

As shown in FIG. 27A, image 2700 may include at least part of a shelf2702. In the example shown, shelf 2702 may be a shelving unit at leastpartially dedicated to dairy products within a retail store. Forexample, shelf 2702 may include milk products 2710, cream products 2720,butter products 2730, and egg products 2740. Shelf 2702 and products2710, 2720, 2730, and 2740 are provided by way of example, and thedisclosed embodiment are not limited to any particular type of displayor product. In some embodiments, server 135 may be configured to analyzeimage to determine a placement of products on the shelf (or part of theshelf). For example, image processing unit 130 may use various imageanalysis techniques to detect products 2710, 2720, 2730, and 2740 onshelf 2702. In some embodiments, the image analysis may includecalculating one or more convolutions of the image, which may facilitatedetermining the placement of products. This may include transposedconvolutions, dilated convolutions, separable convolutions, flattenedconvolutions, grouped convolutions, shuffled group convolutions,pointwise grouped convolutions, or any other form of convolution and maybe performed in single or multiple channels. In one example, at leastone convolution of the first image may be calculated, and the calculatedat least one convolution may be used to determine the first placement ofproducts.

In some embodiments, server 135 may determine that one or more imagesthat are received are insufficient to determine a placement of productson a shelf. For example, some or all of a representation of the shelfmay be obscured in the image. This may be due to a customer, storeassociate, or other individual standing or walking in front of theshelf, a finger of a user appearing the image, an insect on the lens,dust or dirt on the lens, a product or other object blocking the camera,or the like. As another example, the image quality may be insufficientfor detecting at least some of the products. For example, the image maybe too dark or bright, of poor resolution, out of focus, or may haveother characteristics preventing accurate detection of products. Basedon the determination that the image is insufficient, server 135 maycause another image to be captured, which may be analyzed to determinethe placement of products. In some embodiments, this may includegenerating a prompt for an individual, such as a store associate, acustomer, a visitor to the store, a manager, or other individuals tocapture an image. This may include transmitting information to cause anotification to be displayed on a computing device such as devices 145A,145B, 145C and 145D. In some embodiments, this may include generatinginstructions for an image capture device to capture an image or images.For example, this may include generating instructions for one or morecapturing devices 125 associated with a retail store.

Based on the determined placement of objects, server 135 may determine aplanned adjustment to the placement of products on the shelf. Forexample, the adjustment may include removing a particular product from ashelf, adding a new product to a shelf, changing a placement location ofone or more products from the shelf, rotating one or more products,changing a size of a portion of the shelf dedicated to a particularproduct, or any other modifications to product placement or orientation.In some embodiments, planned adjustments may be in reference to a targetplanogram, which may be received by server 135. A target planogramrefers to a planogram that defines a desired or required futureplacement for products on a display structure. For example, the targetplanogram may reflect an updated contractual obligation, a retailer orsupplier preference, a change in seasonal product placements, a remodelor partial remodel of a store or shelving unit, discontinuation of oneor more products, addition of one or more new products, or various otherevents that may affect placement of products on a shelf. As noted above,in some instances, the target planogram may differ significantly fromthe current product placement on a shelf, such that reaching the targetproduct placements may be a significant undertaking for storeassociates. Accordingly, the adjustments to the placement of products onthe shelf determined by system 100 may include smaller, more manageable,steps toward reaching the target planogram.

FIG. 27B illustrates a target planogram 2750 that may be received,consistent with the present disclosure. Target planogram 2750 may beaccessed by server 135 in any suitable manner. In some embodiments,planogram 2750 may be accessed from an external source, such as asupplier, a retail store associated with another retail store, amanagement entity (e.g., a corporate headquarters, a third-partymanagement company, etc.), or any other entities that may provideinstructions regarding product placement. Accordingly, server 135 mayaccess target planogram 2750 through communications network 150 usingnetwork interface 206. In some embodiments, target planogram 2750 may beaccessed from a memory device, such as memory device 226. For example,target planogram may be stored in database 140 or another data structureand may be accessed through memory interface 204. As another example,target planogram 2750 may be accessed through a user interface, such asI/O system 210. For example, a store associate, manager, or other entitymay input a target planogram via a graphical user interface such asthrough touch screen 218. In some embodiments the target planogram maybe determined at least in part by server 135. For example, server 135may be configured to optimize product placement on shelves based onsales data or other information accessed by server 135. The methodsabove are provided by way of example, and the present disclosure is notlimited to any particular means of accessing target planograms.

As illustrated in FIGS. 27A and 27B, target planogram 2750 may representa significant departure from the current placement of products on shelf2702 indicated in image 2700. For example, milk products 2710 may begrouped differently and placed in different locations on shelf 2702,creamer products 2720 may be moved to a different location and reducedin variety, a portion of shelf 2702 dedicated to butter products 2730may be reduced, an additional product 2752 may be added, and eggproducts 2740 may be reorganized. While any one of these changes may beeasily managed by store associates or other workers tasked with makingthe change, collectively these changes may be more daunting. It is to beunderstood that the remodel of shelf 2702 depicted in FIGS. 27A and 27Bis a relatively simplified remodeling task for purposes of illustration.For example, one would recognize that target planograms affecting largerportions of shelving units, or multiple shelving units (and therefore alarger numbers of displayed products) would present increasingly arduoustasks for workers.

Server 135 may determine a plurality of adjustments to the placement ofproducts on shelf 2702, which may be configured to achieve compliancewith target planogram 2750. Accordingly, the adjustments may be a seriesof steps that, if carried out correctly, would result in shelf 2702being in compliance with planogram 2750. For example, the adjustmentsmay focus on particular regions of shelf 2702, particular products orproduct types, particular tasks or types of tasks (e.g., productrelocations, product removals, product additions, changes in size ofareas dedicated for particular products), or any other forms of gradualadjustments. FIGS. 28A and 28B illustrate example adjustments to productplacements, consistent with the present disclosure. For example, FIG.28A illustrates an example adjustment 2800 that may focus onrearrangement of the top portion of shelf 2702. In particular,adjustment 2800 may be an adjustment of the placement of products 2710and 2720 to conform with target planogram 2750. For example, adjustment2800 may include changing the position of milk product 2802 and creamerproduct 2804. Adjustment 2800 may further include removal of creamerproduct 2806 from shelf 2702.

FIG. 28B illustrates another example adjustment 2820 that may focus onplacement of products on another portion of shelf 2702. In particular,adjustment 2820 may be an adjustment to the placement of butter products2730 to conform with target planogram 2750. Adjustment 2820 may includea decrease in size of the portion of shelf 2702 dedicated to products2822 and 2824. Further, adjustment 2820 may include the addition of anew product 2752, as described above. It is to be understood thatadjustments 2800 and 2820 are provided by way of example and variousother forms of adjustments may be determined by server 135. In someembodiments, the adjustments may be smaller or larger in scope thanadjustments 2800 or 2820, depending on the particular implementation ofsystem 100 and/or the specific requirements of the retail store.

Consistent with the disclosed embodiments, server 135 may furtherprovide information configured to cause the adjustments to beimplemented. For example, server 135 may generate instructions or otherforms of information that may indicate that the adjustment is to beperformed. The information may include any combination of alphanumericaltext, symbols, images, graphical representations, and/or other elementsfor indicating the adjustment. In some embodiments, the information maybe one or more text-based instructions to implement the adjustment. Forexample, the information for adjustment 2800 may include text such as“move product SKU #02802 to from position 2 to position 5” (assumingshelf 2702 has a series of designated position numbers and product 2802is associated with a Stock Keeping Unit code of 02802). Various otherexample instructions may be provided depending on a stocking scheme forthe retail environment. In some embodiments, the information may be anintermediate target planogram. For example, adjustment 2800 may be arepresentation of a planogram with the updated locations of products2710 and 2720 as shown in FIG. 28A but with products 2730 and 2740 inthe positions shown in FIG. 27A. Accordingly, server 135 may generate aseries of intermediate planograms for reaching target planogram 2750. Asanother example, the information may include a diagrammaticrepresentation of an adjustment, similar to FIG. 28A, 28B, or 28D(described in further detail below). The information may include variousother forms of representing adjustments, including combination of two ormore forms of instructions.

The information may be provided in any manner that would result inimplementation of the planned adjustment. In some embodiments, theinformation may be presented to store associates of the retail store.For example, server 135 may transmit the information to an associatedevice, such as devices 145C and/or 145D, which may present theinformation to a store associate. In some embodiments, this may includedisplaying the information on a user interface of the device. This mayinclude providing a notification indicating the planned adjustment,displaying images indicating the planned adjustment (e.g., diagrams,planograms, etc.), adding a task associated with the planned adjustmentto a list of tasks to be completed by the associate, adding a calendarentry or other scheduling entry associated with the planned adjustment,or any other manner of presenting the information. The information maybe presented consistent with other example outputs for an employee orassociate of a retail store as described throughout the presentdisclosure (e.g., as shown in FIG. 11D). Additionally or alternatively,the information may be presented audibly to the store associate. Forexample, the device may provide spoken instructions to the associateindicating the planned adjustment. In some embodiments, the indicationmay be an alert indicating that a planogram, list of tasks, or otherstored data has changed, which may prompt the store associate to viewthe data.

As another example, the information may be presented to a manager of theretail store. For example, server 135 may transmit the information to astore management computing device, which may be used to manage planogramcompliance, restocking tasks, or other operations of the retail store.The management device may in turn provide instructions to a storeassociate, for example, by providing instructions via an associatedevice such as devices 145C and/or 145D as described above. In someembodiments, the information may be presented to a manager of the retailstore via the management device and the manager may provide theinstructions to the store associates. For example, the information maybe displayed as part of GUI 1120, as described above with respect toFIG. 11C. In some embodiments, the information may be provided to otherentities associated with the retail store or the displayed products. Forexample, server 135 may transmit the information to a supplier (e.g.,supplier 115A, 115B, or 115 c), to market research entity 110, toanother retail store, or various other entities.

In some embodiments, planned adjustments may be determined such thatthey coincide with or otherwise correspond with other tasks associatedwith shelf 2702. As an example, an adjustment may be planned to coincidewith a restocking task associated with the shelf. If some or all of aproduct needs to be restocked, there may be more space available on theshelf for shifting products around, replacing products, foregoingrestocking of certain products, or other tasks that may be used to carryout a planned adjustment. For example, referring to FIG. 28B, adjustment2820 may be designed to coincide with restocking of products 2822 and/or2824. If the number of products 2822 and 2824 are relatively low, it maybe easier to reduce the size of the portion of shelf 2702 dedicated toproducts 1822 and 2824 and to introduce new product 2752. Further, if astore associate can combine tasks to be completed at the same time, thismay improve efficiency for the associate and the retail store generally.

In some embodiments, the information indicating the planned adjustmentmay be provided based on a timing of a restocking event. For example,server 135 may wait to provide the information until the restocking isneeded. This may be determined based on analyzing the placement ofproducts on shelf 2702 (e.g., by monitoring and analyzing images ofshelf 2702) to detect when a quantity of a product is low. Alternativelyor additionally, the information indicating the planned adjustments maybe presented prior to the restocking event but may indicate that theadjustment should be performed during the next restocking event. Theplanned adjustments may be coordinated with various other events, suchas promotional events, seasonal events, maintenance tasks, cleaningtasks, periods with less customer traffic, or any other events that mayenable the planned adjustment to be carried out more effectively.Integrating the planned adjustments with other events in this manner mayminimize the level of effort required by managers or store associates.

As described above, adjustments 2800 and 2820 may be part of an overallprocedure determined by server 135 for reaching compliance with targetplanogram 2750. In some embodiments, server 135 may determine theoverall procedure initially and may gradually present the adjustments.For example, server 135 may outline a series of planned adjustments forcompliance with target planogram 2750 and may provide informationconfigured to cause the planned adjustments to be implementedindividually or in subgroups of planned adjustments. In someembodiments, the planned adjustments may be associated with target orrequired completion times. For example, the series of plannedadjustments may be scheduled to reach compliance with target planogram2750 by a specified date. The specified target date may be defined basedon contractual obligations (e.g., for planogram compliance, etc.),supplier or retailer preferences, based on predefined deadlines (e.g.,the beginning of summer, the end of a promotion, beginning or end of aproduct being available, etc.), default timelines of the system, orother dates that may affect the placement of products on shelfs.Accordingly, server 135 may provide alerts or reminders for completingthe planned adjustments by the scheduled date. In some embodiments, thepresentation of information indicating the planned adjustments may alsobe scheduled. For example, the incremental planned adjustments may bepresented to store managers or associates at set intervals (e.g., daily,weekly, monthly, etc.), at intervals based on an expected level ofeffort to accomplish the adjustment, or other intervals. In someembodiments, server 135 may await confirmation of completion of aprevious planned adjustment before providing information for asubsequent planned adjustment within the overall plan of adjustments.For example, server 135 may monitor images of shelf 2702, receiveconfirmation information from a manager or associate, or receive sensordata, or otherwise access information that may indicate the adjustmenthas been completed.

In some embodiments, the planned adjustments may be determined and/ormodified dynamically. For example, sever 135 may analyze a first imageto provide information regarding a first adjustment as described above.Then, after the information regarding the first adjustment was provided,subsequent images may be captured and analyzed to determine additionalplanned adjustments. Accordingly, the planned adjustments may notreflect an overall plan determined at one time, but may be determinedprogressively. In some embodiments, this may be based on a status of anexecution of the first adjustment indicated in subsequent images. Forexample, server 135 may receive a second image including arepresentation of shelf 2702 and may determine a second plannedadjustment based on the placement of products indicated in the secondimage. This process may be repeated for subsequent images until anoverall reconfiguration of the shelf is complete (e.g., to comply withtarget planogram 2750). For example, after providing informationindicating adjustment 2800, server 135 may analyze subsequent images toverify that products 2710 and 2720 have been rearranged as indicated byadjustment 2800. If a subsequent image indicates a successful executionof adjustment 2800, server 135 may determine another adjustment based onthe updated placement of products on shelf 2702, such as adjustment2820.

Conversely, the subsequent image may indicate a failed execution ofprevious planned adjustments. A failed execution refers to any productplacement that does not comply with the planned product placement afterthe adjustment. For example, a failed execution may include a failure toattempt to implement the planned adjustment if the products are in thesame position the planned adjustment was based on. As another example,the failed execution may include an incorrect implementation of aplanned adjustment. In this example, a subsequent image may indicatethat products have been repositioned based on the planned adjustment,but at least one product position is incorrect. In some other examples,the subsequent image may indicate a product placement that does notcomply with the planned product placement even after successfulexecution of adjustment 2800, for example due to actions of customers inthe retail store. FIG. 28C illustrates an example image 2840 showing afailed execution of a planned adjustment, or a deviation from thedesired product placement after the planned adjustment due to any otherreason, consistent with the present disclosure. Image 2840 may be animage captured after adjustment 2800 was determined and provided asdescribed above. Image 2840 may indicate that adjustment 2800 wasexecuted incorrectly, or deviated from the desired product placementafter the planned adjustment due to any other reason. For example, mostof products 2710 and 2720 may be positioned correctly, however, product2842 may be in the wrong position on shelf 2702.

Server 135 may take one or more actions in response to a failedexecution identified in a subsequent image or in response to a deviationfrom the desired product placement after the planned adjustment due toany other reason. In some embodiments, this may include generating anotification or other indication of the failed execution. For example,this may include generating a reminder that the planned adjustment stillneeds to be completed, which may be presented to the same entity orperson as the information indicating the planned adjustment (e.g., tothe store manager or store associate). As another example, server 135may cause a notification to be generated for another entity. Forexample, if the original information was presented to a store associate,a notification may be provided to a manager indicating the plannedadjustment has not been completed correctly. The notification may beprovided to other entities, such as another retail store, a supplier, ora market research entity.

In some embodiments, the failed execution or the deviation from thedesired product placement after the planned adjustment due to any otherreason may be used for determining subsequent planned adjustments. Forexample, based on image 2840, server 135 may determine an additionalplanned adjustment to correct the deviation or the failed execution ofadjustment 2800. This may include a planned adjustment to move product2842 to the end of shelf 2702, as shown in FIG. 28A. In someembodiments, the failed execution may be used as feedback regarding thedetermined adjustments. For example, the failure of store associates toimplement planned adjustment 2800 within an expected timeframe or toimplement it correctly may indicate that planned 2800 was too complex ordifficult, was too large in scope, was presented at an inconvenienttime, or other forms of feedback. Accordingly, server 135 may adjustfuture planned actions that are determined. For example, rather thanpresenting planned adjustment 2820, server 135 may present a series ofsmaller, more manageable adjustments.

FIG. 28D illustrates example modified adjustments 2860, 2862, and 2864that may be determined based on subsequent images, consistent with thepresent disclosure. For example, rather than presenting plannedadjustment 2820, server 135 may present planned adjustment 2860, whichmay include reducing the size of the portion of shelf 2702 dedicated toproducts 2822 and 2824, as shown. Planned adjustment 2820 may further bebroken down into adjustment 2862 of relocating the display of product2824 and adjustment 2864 of introducing new product 2752. Server 135 mayevaluate images captured between adjustments 2860, 2862, and 2864 todetermine whether further tuning of the gradual adjustments is needed.Adjustments 2820, 2860, 2862, and 2864 are provided by way of example,and it is to be understood that various other forms of adjustments maybe used, including adjustments with varying degrees of complexity.

Various other forms of feedback may be used by server 135 for tuning orrefining planned adjustments. In some embodiments, the feedback may befeedback provided by a user of system 100, such as user 120. Forexample, a store associate, manager, supplier, market research entity,customer, or other entity associated with system 100 may providefeedback that may indicate one or more properties of the plannedadjustments should be modified. For example, a manager or storeassociate may enter information indicating that a planned adjustment istoo difficult. Accordingly, server 135 may reconfigure the plannedadjustment and/or future planned adjustments to reduce the scope,coincide better with other tasks (e.g., restocking tasks), provide moretime for completing the adjustment, or the like. In some embodiments,the feedback may be indirect. For example, a customer may provideinformation indicating that a product is difficult to find or not in thecorrect location, which may indicate that store associated are havingtrouble keeping up with the planned adjustments.

As another example, feedback may be based on performance metricsassociated with the retail store, the shelf, or products on the shelf.In some embodiments, server 135 may receive an indication of an impactof a planned adjustment. An impact may include any form of measurableresult of the planned adjustment being implemented. In some embodimentsthe impact may include an impact on sales based on the adjustment. Forexample, server 135 may receive information indicating whether anincrease or decrease in sales occurred for a particular product, forproducts on a shelf or portion of a shelf, for products on othershelves, for products in a retail store generally, for products in otherassociated retail stores, or the like. As another example, the impactmay include an engagement of customers with the shelf. For example, thismay include a number of customers that look at or take products from theshelf, a number of customers that purchase products on the shelf, anumber of customers that pass by the shelf, or other forms ofinteraction. Various other metrics may be used, such as movement ofproducts on the shelf (e.g., restocking rates, reorder rates, etc.),overall store execution efficiency, customer satisfaction scores,employee satisfaction scores, employee turnover, or any other metricthat may be associated with a retail store's performance.

FIG. 29 provides a flowchart of an exemplary method for making gradualadjustments to planograms, consistent with the present disclosure.Process 2900 may be performed by at least one processing device ofserver, such as processing device 302, as described above. In someembodiments, some or all of process 2900 may be performed by a differentdevice associated with system 100. In some embodiments, a non-transitorycomputer readable medium may contain instructions that when executed bya processor cause the processor to perform process 2900. Further,process 2900 is not necessarily limited to the steps shown in FIG. 29,and any steps or processes of the various embodiments describedthroughout the present disclosure may also be included in process 2900,including those described above with respect to FIGS. 27A, 27B, 28A,28B, 28C, and/or 28D.

In step 2902, process 2900 may include receiving a first image of atleast part of a shelf. For example, step 2902 may include receivingimage 2700, which may include at least a portion of shelf 2702, asdescribed above. The first image may be captured by at least one imagesensor associated with a retail environment in which the shelf isincluded. For example, the first image may be acquired by at least oneimage sensor mounted adjacent to the shelf on an additional shelf. Asanother example, the first image may be acquired by at least one imagesensor of a personal mobile device. The personal mobile device may beheld by a store associate, customer, visitor, manger, crowdsourcingparticipant, or other individuals as described above. In someembodiments, the first image may be acquired by at least one imagesensor of a robotic device.

In step 2904, process 2900 may include analyzing the first image todetermine a first placement of products on the at least part of theshelf. For example, image 2700 may be analyzed to determine a placementof products 2710, 2720, 2730, and/or 2740. The first placement ofproducts may be determined based on one or more image analysistechniques as described throughout the present disclosure. In oneexample, Step 2904 may include calculating at least one convolution ofthe first image, and may use the calculated at least one convolution todetermine the first placement of products. For example, in response to afirst value of a calculated convolution, step 2904 may includedetermining one placement of products, and in response to a second valueof the calculated convolution, step 2904 may include determining anotherplacement of products. In some embodiments, step 2904 may includedetermining that the first image is insufficient to determine the firstplacement. For example, the first image may include an obstruction ormay be of poor quality (e.g., blurry, out of focus, pixelated, over- orunder-exposed, incomplete or erroneous, etc.) Accordingly, in responseto the determination that the first image is insufficient to determinethe first placement, step 2904 may include causing at least one of astore associate and a robot to capture an additional image of at least aportion of the at least part of the shelf and analyzing the additionalimage to determine the first placement. For example, this may includegenerating instructions for an image capture device to capture theadditional image, or prompting a store associate, customer, manager, orother individual to capture the additional image.

In some embodiments, a machine learning model may be trained usingtraining examples to determine placement of products on at least part ofa shelf from images. An example of such training example may include asample image of a sample shelf, together with a label indicating theplacement of products on the sample shelf. In one example, step 2904 mayinclude using the trained machine learning model to analyze the firstimage received in step 2902 and determining the first placement ofproducts on the at least part of the shelf. In another example, step2912 (described below) may include using the trained machine learningmodel to analyze the second image received in step 2910 and determiningthe second placement of products on the at least part of the shelf.Similar processes may be used in steps 3202 and 3212 as described belowwith respect to FIG. 32. In yet another example, Step 3204 (describedbelow) may use the trained machine learning model to analyze the firstimage received by Step 3202 and determine the first placement ofproducts on the at least part of the shelf. In an additional example,Step 3212 (described below) may use the trained machine learning modelto analyze the second image received by Step 3210 and determine thesecond placement of products on the at least part of the shelf. In someexamples, one or more sensors (such as weight sensors, pressure sensorstouch sensors, etc.) may be positioned between at least part of theshelf and products placed on the at least part of the shelf. In someexamples, data captured using the one or more sensors positioned betweenthe at least part of the shelf and products placed on the at least partof the shelf may be analyzed to determine placement of products on theat least part of the shelf. In one example, a machine learning model maybe trained using training examples to determine placement of products onat least part of a shelf from data captured using one or more sensorspositioned between the at least part of the shelf and products placed onthe at least part of the shelf. An example of such training example mayinclude a sample data captured using one or more sensors positionedbetween a sample shelf and products placed on the sample shelf, togetherwith a label indicating the placement of products on the sample shelf.Additionally or alternatively to the usage of image analysis, step 2904may analyze data captured using one or more sensors positioned betweenthe at least part of the shelf and products placed on the at least partof the shelf to determine the first placement of products on the atleast part of the shelf. Additionally or alternatively to the usage ofimage analysis, Step 2912 (described below) may analyze data capturedusing one or more sensors positioned between the at least part of theshelf and products placed on the at least part of the shelf after thefirst information was provided to determine the second placement ofproducts on the at least part of the shelf. Similar processes may beused in steps 3202 and 3212 as described below with respect to FIG. 32.Additionally or alternatively to the usage of image analysis, Step 3204(described below) may analyze data captured using one or more sensorspositioned between the at least part of the shelf and products placed onthe at least part of the shelf to determine the first placement ofproducts on the at least part of the shelf. Additionally oralternatively to the usage of image analysis, Step 3212 (describedbelow) may analyze data captured using one or more sensors positionedbetween the at least part of the shelf and products placed on the atleast part of the shelf after the first instructions were provided todetermine the second placement of products on the at least part of theshelf.

In step 2906, process 2900 may include determining, based on thedetermined first placement of products, a planned first adjustment tothe determined first placement of products on the at least part of theshelf. For example, this may include determining planned adjustment2800, as described above. In one example, in response to one placementof products determined in step 2904, step 2906 may include determiningone planned adjustment, and in response to another placement of productsdetermined in step 2904, step 2906 may include determining anotherplanned adjustment. The planned first adjustment may include variousactions affecting the position, quantity, orientation, or otherproperties of the physical placement of products on the shelf. Forexample, this may include removing a particular product type from theshelf, adding a new product type to the shelf, changing a placementlocation of a particular product type on the shelf, changing a size of aportion of the shelf dedicated to a particular product type, or similaractions. In some embodiments, the planned first adjustment may beconfigured to coincide with other events. For example, the planned firstadjustment may be configured to be performed as part of a restockingtask.

In step 2908, process 2900 may include providing, based on the plannedfirst adjustment to the determined first placement of products, firstinformation configured to cause the planned first adjustment to thedetermined first placement of products. In some embodiments, providingthe first information may include transmitting instructions to a storeassociate of a retail store in which the shelf is included. For example,this may include transmitting instructions to a computing device of theretail store, such as devices 145C and/or 145D.

In step 2910, process 2900 may include receiving a second image of theat least part of the shelf captured after the first information wasprovided. As with the first image, the second image may be captured byat least one image sensor associated with the retail environment inwhich the shelf is included. For example, the second image may beacquired by at least one image sensor mounted adjacent to the shelf onan additional shelf, at least one image sensor of a personal mobiledevice, or at least one image sensor of a robotic device. In someembodiments, the second image may not necessarily be acquired by thesame device as the first image and may be acquired by a differentdevice.

In step 2912, process 2900 may include analyzing the second image todetermine a second placement of products on the at least part of theshelf. As with the first placement of products, the second placement ofproducts may be determined based on one or more image analysistechniques as described throughout the present disclosure. In oneexample, step 2912 may include calculating at least one convolution ofthe second image, and using the calculated at least one convolution todetermine the second placement of products. Further, step 2912 mayinclude determining that the second image is insufficient to determinethe second placement, causing at least one of a store associate and arobot to capture an additional image of at least a portion of the atleast part of the shelf and analyzing the additional image to determinethe second placement.

In step 2914, process 2900 may include determining, based on the secondplacement of products, a planned second adjustment to the determinedsecond placement of products on the at least part of the shelf. Forexample, this may include determining planned adjustment 2820, asdescribed above. The planned second adjustment may include removing aparticular product type from the shelf, adding a new product type to theshelf, changing a placement location of a particular product type on theshelf, changing a size of a portion of the shelf dedicated to aparticular product type, or similar actions. As with the planned firstadjustment, the planned second adjustment may be configured to beperformed as part of a restocking task or other event.

In some embodiments, the second image may be used to evaluate completionof the planned first adjustment. For example, process 2900 may includeanalyzing the second image to determine a status of an execution of theplanned first adjustment and based on the status of the execution of theplanned first adjustment, determining the planned second adjustment tothe determined second placement of products on the at least part of theshelf. In some embodiments, the status of the execution of the plannedfirst adjustment may include a determined failure of the execution ofthe planned first adjustment. For example, the determined failure may bebased on an incorrect product placement detected in the second image. Inresponse to a determined failure of the execution of the planned firstadjustment, the planned second adjustment may be a smaller adjustmentthan the planned first adjustment and/or to correct the determinedfailure.

According to some embodiments, the planned second adjustment may bedetermined based on feedback from the planned first adjustment. Forexample, process 2900 may further include receiving an indication of animpact of the planned first adjustment to the determined first placementof products. Accordingly, step 2914 may include determining the plannedsecond adjustment to the determined second placement of products on theat least part of the shelf based on the determined second placement ofproducts and the indication of the impact of the planned firstadjustment. For example, the impact may include an impact on sales,impact on engagement of customers with the shelf, impact on movement ofproducts on the shelf, impact on store execution efficiency, or othermetrics as described above. In one example, in response to a firstfeedback, step 2914 may include determining one adjustment, and inresponse to a second feedback, step 2914 may include determining anotheradjustment.

In step 2916, process 2900 may include providing, based on thedetermined planned second adjustment to the determined second placementof products, second information configured to cause the planned secondadjustment to the determined second placement of products. As with thefirst information, providing the second information may includetransmitting instructions to a store associate of a retail store inwhich the shelf is included. This may include providing the informationto the same device as the first information, or may include providingthe information to a different device.

In some embodiments, process 2900 may include one or more steps inaddition to those shown in FIG. 29. For example, as described above,process 2900 may be performed to reach compliance with a targetplanogram. Accordingly, process 2900 may further include receiving atarget planogram, such as target planogram 2750. Accordingly,determining the planned first adjustment to the determined firstplacement of products on the at least part of the shelf may be based onthe target planogram and the planned first adjustment may be configuredto be a step in changing the determined first placement of products onthe at least part of the shelf towards the target planogram. Similarly,determining the planned second adjustment to the determined secondplacement of products on the at least part of the shelf may be based onthe target planogram and the planned second adjustment may be configuredto be an additional step in changing the determined second placement ofproducts on the at least part of the shelf towards the target planogram.In some embodiments, process 2900 may further include confirmingcompliance with the target planogram has been met. For example, process2900 may include receiving a third image of the at least part of theshelf captured after the second information was provided and analyzingthe third image to determine compliance with the target planogram.

As discussed above, the disclosed systems may be used for managingmodifications to planograms. For example, when a target planogram isknown in advance, the disclosed systems may break a large planogramremodeling task into small manageable tasks, thereby reducing the burdenon store associates. In some embodiments, the disclosed systems may beused to optimize the resulting planogram. For example, the system maysuggest an incremental planogram update, as described above, and mayassess a result or effect of the incremental update. This assessment mayprovide insight for later incremental updates. In some embodiments, atarget planogram may not be identified in advance, and through assessingincremental updates, an optimized target planogram may be achieved.Alternatively, or additionally, even when a target planogram isestablished, the target planogram may be altered or adjusted through theoptimization process. The disclosed embodiments therefore provide, amongother advantages, improved efficiency, convenience, and functionalityover prior art planogram compliance and inventory management systems.

As described above, the disclosed embodiments may include receivingimages of at least part of a shelf. FIG. 30A illustrates an exampleimage 3000 including at least part of a shelf 3002, consistent with thepresent disclosure. Consistent with the present disclosure, shelf 3002may be the same as shelf 2702 described above. Accordingly, any of themethods or features described above with respect to FIGS. 27A, 27B, 28A,28B, 28C, 28D and/or 29 may also apply to shelf 3002. Image 3000 may beacquired by an image sensor included in the retail store, such as imagesensor 310 included in capturing device 125. The capturing device maytake various forms or may be mounted in various locations, as describedthroughout the present disclosure. As described above with respect toimage 2700, image 3000 may be captured by a capturing device mountedadjacent to shelf 3002, by a handheld device, by a robotic device, orany other form of capturing device described herein.

Image 3000 may include at least part of shelf 3002, as shown. In thisexample, shelf 3002 may be a shelving unit at least partially dedicatedto dairy products within a retail store. For example, shelf 3002 mayinclude milk products 3010, cream products 3020, butter products 3030,and egg products 3040. Shelf 3002 and products 3010, 3020, 3030, and3040 are provided by way of example, and the disclosed embodiment arenot limited to any particular type of display or product. In someembodiments, server 135 may be configured to analyze an image todetermine a placement of products on the shelf (or part of the shelf),as described above. For example, image processing unit 130 may usevarious image analysis techniques to detect products 3010, 3020, 3030,and 3040 on shelf 3002. In some embodiments, the image analysis mayinclude calculating one or more convolutions of the image, which mayfacilitate determining the placement of products. For example, inresponse to a first value of a calculated convolution of at least partof the image, one placement of products on shelf 3002 may be determined,and in response to a second value of the calculated convolution, anotherplacement of products on shelf 3002 may be determined. This may includetransposed convolutions, dilated convolutions, separable convolutions,flattened convolutions, grouped convolutions, shuffled groupconvolutions, pointwise grouped convolutions, or any other form ofconvolution and may be performed in single or multiple channels.

As described above, server 135 may determine a planned adjustment to theplacement of products on the shelf. For example, the adjustment mayinclude removing a particular product from a shelf, adding a new productto a shelf, changing a placement location of one or more products fromthe shelf, rotating one or more products, changing a size of a portionof the shelf dedicated to a particular product, or any othermodifications to product placement or orientation.

FIG. 30B illustrates an example adjustment 3050 that may focus onrearrangement of the top portion of shelf 3002, consistent with thepresent disclosure. In particular, adjustment 3050 may be an adjustmentof the placement of products 3010 and 3020. For example, adjustment 3050may include rearranging the positions of products 3010 and removingproducts 3020 from the shelf. Adjustment 3050 may further includeexpanding the portion of shelf 3002 dedicated to product 3012. In someembodiments, adjustment 3050 may be determined in reference to a targetplanogram. For example, the target planogram may dictate that products3010 and 3020 be arranged as would result from implementing adjustment3050.

Alternatively, or additionally, adjustment 3050 may be determinedwithout reference to a particular planogram. In these embodiments, theadjustments may be determined in various ways. In some embodiments,adjustment 3050 may be determined based on one or more rules. Forexample, a rule may be defined to perform actions on particular producttypes, such as group like products together, move products of aparticular type to a certain location, increase or decrease spacededicated to particular product types, or the like. As used herein, aproduct type refers to any form of classification or category ofproducts. In some embodiments, the rule may be based on sales data,inventory data, or other forms data. Accordingly, the particular producttype may refer to products having a certain popularity, products incertain price ranges, products with certain inventory levels, or thelike. Therefore, a rule may be defined to move popular products to anaverage eye level for customers, or to decrease the portion of a shelfdedicated to items for which an inventory amount is relatively low. Therules described herein are provided by way of example, and one skilledin the art would recognize that many other types of rules could bedefined.

Various other methods for determining adjustment 3050 may be used. Insome embodiments, adjustment 3050 may be determined at least partiallybased on an input from a user. For example, a store associate, manager,customer, secret shopper, or other entity may provide input that maysuggest an adjustment to be implemented. The input may be a suggestiondefining adjustment 3050 (e.g., a request to make a particular productadjustment), or may be information from which adjustment 3050 isderivable (e.g., a survey result, a customer complaint, an employeerequest, etc.). In some embodiments, adjustment 3050 may be random orsemi-random. For example, server 135 may suggest random incrementaladjustments and may gauge an effect of the adjustment, as described ingreater detail below. In some embodiments, the adjustment may be basedon other data, such as results from other products, adjustments made inother retail locations, or the like.

Instructions to implement adjustment 3050 may be provided, consistentwith the embodiments disclosed above. For example, server 135 maygenerate instructions or other forms of information that may indicatethat the adjustment is to be performed. The information may include anycombination of alphanumerical text, symbols, images, graphicalrepresentations, and/or other elements for indicating the adjustment.For example, the information may include text-based instructions, anintermediate target planogram, a diagrammatic representation of anadjustment, or various other means for representing adjustments,including combination of two or more forms of instructions.

Further, as described above, the information may be provided in anymanner that would result in implementation of the planned adjustment. Insome embodiments, the information may be presented to store associatesof the retail store. For example, server 135 may transmit theinformation to an associate device, such as devices 145C and/or 145D,which may present the information to a store associate. This may includedisplaying providing a notification indicating the planned adjustment,displaying images indicating the planned adjustment (e.g., diagrams,planograms, etc.), adding a task associated with the planned adjustmentto a list of tasks to be completed by the associate, adding a calendarentry or other scheduling entry associated with the planned adjustment,or any other manner of presenting the information. Additionally oralternatively, the device information may be presented audibly to thestore associate. As another example, the information may be presented toa manager of the retail store. For example, server 135 may transmit theinformation to a store management computing device, which may be used tomanage planogram compliance, restocking tasks, or other operations ofthe retail store. The management device may in turn provide instructionsto a store associate. In some embodiments, the information may bepresented to a manager of the retail store via the management device andthe manager may provide the instructions to the store associates (e.g.,through associate devices).

In some embodiments, the disclosed embodiments may further includereceiving an additional image to assess or confirm implementation of anadjustment. FIG. 31A illustrates an additional example image 3100captured after instructions for implementing an adjustment weregenerated, consistent with the present disclosure. Similar to image3000, image 3100 may be acquired by an image sensor included in theretail store, such as image sensor 310 included in capturing device 125.Image 3100 may include at least a portion of shelf 3002, as shown. Inthis example, image 3100 may be captured after instructions to implementadjustment 3050 have been provided. Accordingly, image 3100 may be usedto verify whether adjustment 3050 has been implemented yet or has beenimplemented correctly. Using one or more of the various image processingtechniques described herein (e.g., with respect to images 2700, 2800,and/or 3000), server 135 may identify products in image 3100 anddetermine their placements relative to shelf 3002. The placement ofproducts in image 3100 may be compared to the target placement definedby adjustment 3050 to determine whether adjustment 3050 was properlyexecuted. In some embodiments, server 135 may generate furtherinstructions to perform an adjustment to the placement of products onshelf 3002 until the target placement of products defined in adjustment3050 is reached. Alternatively, or additionally, server 135 may takeother actions such as generating an alert or notification thatadjustment 3050 has not been implemented. Server 135 may continue tocapture images periodically until adjustment 3050 has been performed. Insome embodiments, server 135 may proceed with further analysis describedbelow despite the placement of products not matching with adjustment3050. In the example shown in FIG. 31A, the product placements areconsistent with adjustment 3050.

Server 135 may analyze an impact of the resulting placement of productsidentified in an image, such as image 3100. As noted above, thisplacement may reflect adjustment 3050 having been performed or may bedifferent than the target positions defined by adjustment 3050. Server135 may receive data indicative of an impact of the placement ofproducts shown in image 3100 and may analyze this data to determine theimpact. This impact may be used to determine subsequent plannedadjustments for the products. As used herein, an impact refers to anyeffect or result that may be attributable to placement of products on ashelf. As described in further detail below, the impact may be reflectedin product or store sales performance, customer satisfaction, employeemorale, employee or store efficiency, or any other metrics related to aproduct or retail environment that may be affected.

FIG. 31B illustrates example data that may indicate an impact of anadjusted product placement, consistent with the present disclosure. Inparticular, server 135 may access data such as sales data 3112,inventory data 3114, image data 3116, and/or sensor data 3118. Thevarious forms of data may be received or accessed from a data source3110. In some embodiments, data source 3110 may be a database (such asdatabase 140, an external database, a cloud-based data structure, etc.),as shown in FIG. 31B. Alternatively, or additionally, data source 3110may be a device included or in communication with server 135. Forexample, data source 3110 may include a sensor positioned in the store,an image capture device (e.g., capturing device 125), a computing device(e.g., device 145A, 145B, 145C, and/or 145D), or the like. In someembodiments data source 3110 may include multiple data sources. Theconfiguration of data source 3110 may depend on the particularimplementation of the disclosed embodiments and the type of data beinganalyzed.

Sales data 3112 may include any data associated with the sale ofproducts. For example, sales data 3112 may include monetary salesinformation (e.g., revenue or profit data, product pricing), salesvolume (e.g., a number of products sold, sales growth, etc.), customerreviews, customer traffic (e.g., a number of customers visiting a store,a number of customers visiting shelf 3002, a number of customers lookingat a particular product, etc.), or any other data related to productsales. Sales data 3112 may be data pertaining to a particular store,such as the retail store in which shelf 3002 is located. For example,server 135 may analyze sales data 3112 to determine whether reorganizingproducts 3010 as suggested in adjustment 3050 resulted in an increase insales volume for these products. Server 135 may also analyze the sale ofother products (e.g., products not directly repositioned or adjusted inadjustment 3050) to determine whether adjustment 3050 had a positive ornegative impact on sales for the other products. Additionally, oralternatively, sales data 3112 may include data from other stores, suchas a global sales data for an organization, sales data for nearbystores, competitor sales data, or other data. Accordingly, sales dataassociated with shelf 3002 (e.g., for products on shelf 3002, the retailstore including shelf 3002, etc.) may be compared with other sales datato determine an effect of adjustment 3050. For example, even if sales ofa particular product decrease for the store that includes shelf 3002,the decrease may be less than a global decrease in sales for the productindicated by sales data 3112, which may indicate a relatively positiveeffect of adjustment 3050. The sales data and types of analysis areprovided by way of example, and one skilled in the art would recognizemany other forms of sales data and analysis of the data could be used.

Inventory data 3114 may include any form of data associated with aninventory of products. In some embodiments, inventory data 3114 mayinclude a number of products stored in a store room or other storagearea associated with a retail store. For example, inventory data 3114may refer to an inventory of products 3010, 3020, 3030, and/or 3040included in a storage area of the retail store including shelf 3002. Inthis context, a storage area may refer to any location in which aplurality of products may be stored. The storage area may be a storageroom, a portion of a shelving unit within the retail store dedicated tostorage (e.g., a top shelf, etc.), or any other suitable storagelocation. In other embodiments, inventory data 3114 may refer tooff-site storage, for example, in a warehouse, in a delivery truck, orthe like. In some embodiments, inventory data 3114 may include datapertaining to inventory management. For example, this may include aspeed or efficiency at which products are restocked by store associates.This inventory data may provide insight into an impact of adjustment3050. For example, server 135 may determine how the adjusted productplacement affects how quickly items are restocked, a frequency at whichproducts are re-ordered, an amount of a particular product in inventory,or the like.

Image data 3116 may include any information that may be attained fromimages. For example, this may include images of shelf 3002, such asimage 3100 described above. In some embodiments, server 135 may monitorshelf 3002 to assess an impact of adjustment 3050. For example, this mayinclude periodically analyzing captured images, analyzing a videostream, generating requests to capture images, or any other way in whichan impact may be determined from image data. The images may be capturedby the same image capture device that captured image 3100, or may be adifferent device. For example, server 135 may prompt a store associateto capture an image of shelf 3002, or a customer may capture an image ofshelf 3002 using output device 145D.

Various types of information may be obtained from the images. In someembodiments, server 135 may determine placement or movement of items onthe shelf. For example, if items are frequently taken from the shelf (orreturned to the shelf) it may indicate an effect on the customer'simpression of or desire for the products. Other examples may include howlong a particular item stays on the shelf, planogram compliance (e.g.,how accurately items are restocked according to the current planogram),how often products are restocked, whether customers return the productto the same location they took it from, how long a portion of a shelf isempty, or other information that may be gathered about placement ofproducts on the shelf. The images may also detect an interaction betweencustomers and the shelf or products. For example, the images may be usedto detect, customers touching the products, customers looking at theproducts (e.g., which portion of shelf 3002 customers look at, etc.),customers walking by the products, customer traffic or movement througha store, or other interactions. Server 135 may similarly detectinteractions with other entities, such as managers, store associates,secret shoppers, third party vendors, or the like. In some embodiments,the images may not include shelf 3002 but may be images of otherrelevant areas. For example, this may include an entrance to a retailstore (e.g., detecting customer traffic), a stock room (e.g., fortracking inventory and/or product movement), a break room (e.g.,detecting employee morale through facial expressions), or various otherimage data.

Sensor data 3118 may include any information that may be gathered usingsensors. In this context, sensors may include any form of device orinstrument for measuring data associated with products. In on example,such sensors may be configured to be placed between a shelf and productsplaced on the shelf. In some examples, embodiments may use one or moreof pressure sensors, weight sensors, light sensors, resistive sensors,capacitive sensors, inductive sensors, vacuum pressure sensors, highpressure sensors, conductive pressure sensors, infrared sensors,photo-resistor sensors, photo-transistor sensors, photo-diodes sensors,ultrasonic sensors, microphones, motion sensors, or the like, asdescribed throughout the present disclosure. These sensors may be usedto detect placement of products on a shelf, products being removed orreturned from the shelf, customer traffic near a shelf, customer oremployee speech (indicating customer experience, employee morale, etc.),restocking events, or other events or characteristics that may bedetected through sensors. While various events or characteristics may bedescribed as being determined from one type of data, it is understoodthat they may be similarly obtained from other types of data, includingsales data 3112, inventory data 3114, image data 3116, and sensor data3118. For example, a rearrangement event (i.e., products beingrearranged on a shelf) may be determined from captured images but mayalso be determined from sensor data, such as weight or pressure sensorsindicating products have been removed and/or added to the shelf.Further, while sales data 3112, inventory data 3114, image data 3116,and sensor data 3118, are provided by way of example, various other datamay be analyzed, including customer experience data (e.g., onlinereviews, survey results, etc.), employee feedback, manager feedback, orvarious other forms of data.

Consistent with the disclosed embodiments, server 135 may determinesubsequent adjustments to the placement of products on shelf 3002 basedon the determined impact (or an absence of an impact). FIG. 31Cillustrates another example adjustment 3140 that may be generated basedon a determined impact, consistent with the disclosed embodiments. Asshown in FIG. 31C, adjustment 3140 may include a recommendation toincrease a portion of shelf 3002 dedicated to product 3142. This may bebased on an impact detected in response to adjustment 3050, as describedabove. For example, server 135 may determine a positive impact resultingfrom increasing a portion of shelf 3002 dedicated to product 3012 asimplemented in adjustment 3050. Accordingly, adjustment 3140 may attemptto obtain similar positive results with respect to product 3142.Adjustment 3140 may be determined based on the impact of adjustment 3050in various other ways, such as avoiding a negative impact fromadjustment 3050, trying an alternative approach for achieving a resultarising from or sought out by adjustment 3050, counteracting an impactfrom adjustment 3050, augmenting an impact of adjustment 3050, or thelike.

In some embodiments, one or more of the steps described above may berepeated to reach an optimized placement of products on the shelf. Forexample, server 135 may iterate this process multiple times bydetermining an adjustment, analyzing an impact of the adjustment, andgenerating subsequent adjustments based on the determined impact.Accordingly, this iterative process may be used to test incrementalchanges to a planogram and adjust subsequent changes to reach anoptimized planogram (or to make continuous or periodic improvements overtime). As a result, store associates may be tasked with makingrelatively small modifications to the placement of products on the shelfthat are tailored toward achieving a desired result. This may avoidhaving store associates make large changes to the placement of productsthat may or may not be effective.

Further, in some embodiments, resulting impacts from multipleadjustments maybe used to generate a target planogram. For example,impacts resulting from two or more adjustments may be analyzed togetherto determine an overall desired or target planogram. Accordingly, asample of impacts resulting from adjustments to planograms may beanalyzed and extrapolated into an overall planogram. This desiredplanogram may be output as a result of the optimization process. In someembodiments, the desired planogram may be determined using anoptimization algorithm. For example, the impacts from a series ofadjustments may be input into a greedy algorithm such that an optimalsolution may be reached. In some embodiments, server 135 may accessdetermined impacts from multiple products, shelves, or stores over time,which may be input into an optimization algorithm to achieve a morerobust solution.

FIG. 32 provides a flowchart of an exemplary method for testing ofplanograms, consistent with the present disclosure. Process 3200 may beperformed by at least one processing device of a server, such asprocessing device 302, as described above. In some embodiments, some orall of process 3200 may be performed by a different device associatedwith system 100. In some embodiments, a non-transitory computer readablemedium may contain instructions that when executed by a processor causethe processor to perform process 3200. Further, process 3200 is notnecessarily limited to the steps shown in FIG. 32, and any steps orprocesses of the various embodiments described throughout the presentdisclosure may also be included in process 3200, including thosedescribed above with respect to FIGS. 30A, 30B, 31A, 31B, and 31C.

In step 3202, process 3200 may include receiving a first image of atleast part of a shelf. For example, step 3202 may include receivingimage 3000, which may include at least a portion of shelf 3002, asdescribed above. The first image may be captured by at least one imagesensor associated with a retail environment in which the shelf isincluded. For example, the first image may be acquired by at least oneimage sensor mounted adjacent to the shelf on an additional shelf. Asanother example, the first image may be acquired by at least one imagesensor of a personal mobile device. The personal mobile device may beheld by a store associate, customer, visitor, manger, crowdsourcingparticipant, or other individuals as described above. In someembodiments, the first image may be acquired by at least one imagesensor of a robotic device.

In step 3204, process 3200 may include analyzing the first image todetermine a first placement of products on the at least part of theshelf. For example, image 3000 may be analyzed to determine a placementof products 3010, 3020, 3030, and/or 3040. The first placement ofproducts may be determined based on one or more image analysistechniques as described throughout the present disclosure. In someembodiments, step 3204 may include determining that the first image isinsufficient to determine the first placement and causing a storeassociate, manager, robot, or other entity to capture an additionalimage of at least a portion of the at least part of the shelf. In someembodiments, some or all of process 3200 may be performed in conjunctionwith process 2900 described above. For example, step 3204 may use theanalysis of step 2904 described above to analyze the first image anddetermine the first placement of products.

In some embodiments, a machine learning model may be trained usingtraining examples to determine placement of products on at least part ofa shelf from images, as described above with respect to FIG. 29. Forexample, step 3204 may include using the trained machine learning modelto analyze the first image received in step 3202 and determining thefirst placement of products on the at least part of the shelf. Asanother additional example, step 3212 (described below) may includeusing the trained machine learning model to analyze the second imagereceived in step 3210 and determining the second placement of productson the at least part of the shelf. In some embodiments data capturedusing one or more sensors positioned between the at least part of theshelf and products placed on the at least part of the shelf may beanalyzed to determine placement of products on the at least part of theshelf. For example, in addition to or as an alternative to the usage ofimage analysis, step 3204 may include analyzing data captured using oneor more sensors positioned between the at least part of the shelf andproducts placed on the at least part of the shelf to determine the firstplacement of products on the at least part of the shelf. Further, inaddition to or as an alternative to the usage of image analysis, step3212 (described below) may include analyzing data captured using one ormore sensors positioned between the at least part of the shelf andproducts placed on the at least part of the shelf after the firstinstructions were provided to determine the second placement of productson the at least part of the shelf.

In step 3206, process 3200 may include determining, based on thedetermined first placement of products, a planned first adjustment tothe determined first placement of products on the at least part of theshelf. For example, this may include determining adjustment 3050, asdescribed above. The planned first adjustment may include variousactions affecting the position, quantity, orientation, or otherproperties of the physical placement of products on the shelf. Forexample, this may include removing a particular product type from theshelf, adding a new product type to the shelf, changing a placementlocation of a particular product type on the shelf, changing a size of aportion of the shelf dedicated to a particular product type, or similaractions. In some embodiments, the planned first adjustment may beconfigured to coincide with other events, as described in further detailabove. For example, the planned first adjustment may be configured to beperformed as part of a restocking task or similar operational task. Inone example, in response to one placement of products determined in step3204, step 3206 may include determining one adjustment, and in responseto another placement of products determined in step 3204, step 3206 mayinclude determining another adjustment.

In step 3208, process 3200 may include generating first instructions toimplement the planned first adjustment. For example, step 3208 mayinclude generating instructions to implement adjustment 3050, asdescribed above. In some embodiments, providing the first instructionsmay include transmitting instructions to a store associate of a retailstore in which the shelf is included. For example, this may includetransmitting instructions to a computing device of the retail store,such as devices 145C and/or 145D.

In step 3210, process 3200 may include receiving a second image of theat least part of the shelf captured after the first instructions weregenerated. For example, step 3210 may include receiving image 3100,which may be captured after instructions to implement adjustment 3050were generated. As with the first image, the second image may becaptured by at least one image sensor associated with the retailenvironment in which the shelf is included. For example, the secondimage may be acquired by at least one image sensor mounted adjacent tothe shelf on an additional shelf, at least one image sensor of apersonal mobile device, or at least one image sensor of a roboticdevice. In some embodiments, the second image may not necessarily beacquired by the same device as the first image and may be acquired by adifferent device.

In step 3212, process 3200 may include analyzing the second image todetermine a second placement of products on the at least part of theshelf. As with the first placement of products, the second placement ofproducts may be determined based on one or more image analysistechniques as described throughout the present disclosure. Further, step3212 may include determining that the second image reflects a failedexecution of the planned first adjustment, as described above.Accordingly, step 3212 may include generating additional instructions toimplement the planned first adjustment to the first placement ofproducts. In some embodiments, step 3212 may use step 2912 (describedabove with respect to FIG. 29) to analyze the second image and determinethe second placement of products.

In step 3214, process 3200 may include receiving an indication of animpact of the second placement of products. The indication of the impactmay be received in various ways. For example, the indication of theimpact may be based on sensor data, such as an analysis of input from apressure sensor positioned on the at least part of the shelf, a weightsensor connected to the at least part of the shelf, a touch sensorpositioned on the at least part of the shelf, a proximity sensorpositioned on or near the at least part of the shelf, or various otherforms of sensors. In some embodiments, the sensor may be an imagesensor. Accordingly, the impact of the second placement of products maybe based on an analysis of images of the at least part of the shelf.Based on the sensor data, various events or characteristics of theproducts on the shelf may be determined, which may indicate the impact.For example, the impact of the second placement of products may be basedon a product being at least one of returned to or taken from the atleast part of the shelf, a restocking event associated with the at leastpart of the shelf, a rearrangement event of the at least part of theshelf, or the like. As another example, the indication of the impact ofthe second placement of products may be based on an analysis of salesdata, as described above.

In step 3216, process 3200 may include determining a planned secondadjustment to the second placement of products on the at least part ofthe shelf. The planned second adjustment may be determined based on theimpact determined in step 3214. For example, step 3216 may includedetermining adjustment 3140, which may be based on an impact ofadjustment 3050 as described above. The planned second adjustment may bedetermined to achieve a similar impact, avoid a similar impact, augmentthe impact, counteract or reduce the impact, determine whether otheradjustments result in similar impacts, or other desired results that maybe informed by the impact. Step 3216 may include the use of any form ofoptimization algorithm to determine a planned second adjustment to thesecond placement of products on the at least part of the shelf, such asgenetic algorithms, gradient descent algorithms, and so forth. In oneexample, a gradient of the impact in a mathematical space of planogramsmay be estimated based on the first placement of products determined instep 3204, the second placement of products determine in step 3212, andthe impact of step 3214. Step 3216 may include using the estimatedgradient to determine the planned second adjustment to the secondplacement of products on the at least part of the shelf. In someexamples, the second adjustment may be expected to cause a positiveimpact, for example based on the impact of the second placement ofproducts.

In step 3218, process 3200 may include generating second instructions toimplement the planned second adjustment. For example, as with the firstadjustment, step 3218 may include transmitting instructions to a storeassociate of a retail store in which the shelf is included. For example,this may include transmitting instructions to a computing device of theretail store, such as devices 145C and/or 145D.

In some embodiments, process 3200 may further include variousoptimization steps for optimizing product placement on the shelf. Insome embodiments, this may include repeating one or more steps describedabove as an iterative process. For example, process 3200 may includereceiving a third image of the at least part of the shelf captured afterthe second instructions were generated and analyzing the third image todetermine a third placement of products on the at least part of theshelf, the third placement of products resulting from the planned secondadjustment. Process 3200 may then include receiving an indication of animpact of the third placement of products. Further, a planned thirdadjustment may be determined based on the impact of the third placementof products, and so on.

In some embodiments, multiple impacts may be analyzed together todetermine a desired planogram, as described above. For example, process3200 may include using the second placement of products, the indicationof the impact of the second placement of products, the third placementof products and the indication of the impact of the third placement ofproducts to determine a desired planogram. In some embodiments,determining the desired planogram includes using an optimizationalgorithm, such as a greedy algorithm. Process 3200 may further includegenerating information related to the desired planogram. For example,the information related to the desired planogram may include arecommendation to implement the desired planogram and process 3200 mayfurther include transmitting the recommendation to a store associate ofa retail store in which the shelf is included. In some embodiments, theinformation related to the desired planogram includes an expected impactof the desired planogram. For example, the information related to thedesired planogram may indicate the desired planogram is optimized toimprove sales for a retail store, improve sales for a particular productor product type, improve efficiency for restocking products, increaseplanogram compliance, increase or reduce inventory of a product instorage, or other desired outcomes. In some embodiments, process 3200may include generating multiple desired planograms with differentdesired outcomes. For example, the determined impacts may be input intomultiple optimization algorithms (or a single optimization algorithmdesigned to provide multiple solutions) to generate multipleoptimization outcomes. Accordingly, a manager, store associate, or otherentity may be presented with multiple options for planograms withdifferent outcomes or benefits.

In some situations, a person may use an application, such as on asmartphone, to perform a task in a retail store or other environment,such as restocking an item, rearranging items, adjusting a price, andother tasks described herein. In some situations, the possibility of areward may be offered to the person to incentivize performance of thetask, and performance to a certain degree of quality. In some cases, aperson may accept a task at a user device, and may transmit confirmationof task completion to another device, allowing for remote tracking oftask performance. In these scenarios, it can also be advantageous to useimage processing techniques to determine whether a task has beenperformed, and whether the task has been performed to a certain degreeof quality. Information determined through image analysis may then beused to determine an appropriate reward for performance of the task,which may be awarded to the person the user device. The embodimentsdescribed below, operable with other embodiments disclosed herein,discuss these techniques and how they may be implemented to provide tasktracking and reward correlation.

FIG. 33 depicts an exemplary shelf view 3300, which may be representedin a captured image (for example, in an image captured in response to areward offering). In some embodiments, shelf view 3300, or anothersimilar shelf view, may be represented by one or more images, which mayhave been captured by a capturing device 125, output device 145, or anyother device connected to or including an image sensor. In someembodiments, an image of shelf view 3300 may be captured by an imagesensor after a notification of a task request (which may indicate apotential for a reward) was delivered to the device, or after at least aselected time passed since such notification was delivered to thedevice. In one example, the selected time may be selected based on oneor more of a type of the task, a parameter of the task, a productcorresponding to the task, a location corresponding to the task, acurrent time, a person corresponding to the device, and so forth.Additionally or alternatively, an image of shelf view 3300 may becaptured by an image sensor after a determination that a device havingthe image sensor is within a predetermined distance of, and/or is beyonda predetermined distance of (e.g., to capture a full shelf or productgrouping), at least one shelf and/or at least one product. Additionallyor alternatively, an image of shelf view 3300 may be captured after anindication of a completeness of the task is received (for example, fromthe device, from a different device, from a user, and so forth). Anyobject, structure, or other physical thing in exemplary shelf view 3300may be captured in an image and analyzed, as described below, such aswith respect to processes 3400 and/or 3500.

Shelf view 3300 may include one or more shelves 3302, which may beaffixed to a shelving unit, wall, or other structure. While shelves 3302are shown here as an example, it is appreciated that other structuresfor holding products may be shown instead, such as displays, containers,drawer units, bin units, baskets, pegboards, etc. For example, shelves3302 may hold a product that may be associated with a task for which areward is being offered. In some embodiments, the product and/or taskmay be associated with a retail store. In some embodiments, shelves 3302may be situated behind a door (e.g., a refrigerator or freezer door) orother barrier. In some embodiments, shelves 3302 may support one or moreproducts, which may be products of a same or different type.

Shelf view 3300 may include a number of products, such as product 3304a, product 3304 b, product 3308 a, and/or product 3308 b. In someembodiments, some products in shelf view 3300 may be associated with atask (e.g., for which a reward is offered), and others may not beassociated with the task (and may or may not be associated with othertasks). For example, product 3304 a and product 3304 b may be associatedwith a task, but products 3308 a and 3308 b may not be associated withthe task. Any combination of products in a retail store is possible. Insome embodiments, a shelf 3302 may support products of a same type. Forexample, product 3304 a and product 3304 b may be of a same producttype, and product 3308 a and product 3308 b may be of a same producttype. A same product type may be considered any combination ofcommonalities between products, such as a same product identifier (e.g.,Universal Product Code, or UPC, International Article Number, or IAN,European Article Number, or EAN, etc.), a same brand, a same productcategory (e.g., cosmetics, food, electronics, etc.), and/or a samepurpose (e.g., electrical cord for charging a USB device, a cake mix, awrench, etc.).

In some embodiments, a product, such as product 3304 a, may have aproduct label 3306, which may be associated with a task (e.g., for whicha reward is offered). In the example of FIG. 33, product label 3306 maybe attached to the product. In other examples, product label 3306 may beattached to a part of a shelf or to other objects (such as displays,containers, etc.) in the retail store that is associated with theproduct (e.g., stocks the product). A product label may include anyinformation that may distinguish one product from another, which adevice having at least one processor may use to differentiate betweenrelevant and irrelevant information (as discussed below). For example, aproduct label may include any combination of a logo, a barcode, a quickresponse (QR) code, a word, a character, a shape, a color, a placementlocation of the label relative to the product, or any other informationdetectable by an image sensor.

FIG. 34 depicts a flowchart of exemplary process 3400 for providing areward based on image analysis. For purposes of illustration, in thefollowing description, reference is made to certain components of system100. For example, any combination of steps of process 3400 may beperformed by at least one processor of a device such a handheld device(e.g., a smartphone, a tablet, a mobile station, a personal digitalassistant, a laptop, and more), a wearable device (e.g., smart glasses,a smartwatch, a clip-on camera), and/or server. Examples of such devices(e.g., capturing device 125, server 135) are described above. It will beappreciated, however, that other implementations are possible and thatother components may be utilized to implement the exemplary process3400. It will also be readily appreciated that the illustrated methodcan be altered to modify the order of steps, repeat steps, delete steps,or further include additional steps. While certain aspects are describedin the context of a retail store, it is appreciated that any or all ofthe steps described could be implemented in other environments, such asa warehouse, fulfillment center, stockpile, or other place where a usermay attempt to complete a task related to a product.

At step 3402, at least one processor (e.g., a processor of an imageprocessing unit 130) may receive an indication that a person completed atask corresponding to at least one shelf in a retail store. In someembodiments, an indication may be transmitted from a user device (e.g.,capturing device 125) in response to an image captured by the userdevice and/or a user input (e.g., a user interface selection) receivedat the user device. For example, the indication may be based on an inputfrom the person. In some embodiments, the person may be an employee ofthe retail store, may be a non-employee of the retail store, may be acustomer of the retail store, may be a store associate of the retailstore, may be a visitor of the retail store, and so forth. The personmay perform any number of tasks (only one task, two tasks, less thanfive tasks, etc.).

In some embodiments, a task may include, without limitation, at leastone of: a restocking of a product associated with the at least oneshelf, a correction of product facings at the at least one shelf,removing a product from the at least one shelf, changing a price of atleast one product, or changing an orientation of at least one product onthe at least one shelf (e.g., to create a planogram of products). A taskmay also include, without limitation, at least one of positioning orremoving a promotional sign from at least one shelf (or from a vicinityof a shelf). Other non-limiting examples of tasks are described herein.In some embodiments, information associated with a task may betransmitted to a user device, such as one or more locations, aisleidentifiers, products, shelf levels, shelf heights, product identifiers,product images, product sizes, product placements, and the like. Forexample, the at least one processor may transmit a location and productidentifier associated with a first task and a location and productidentifier associated with a second task to a user device. Of course,any number of tasks and combinations of task information may betransmitted to a user device, including tasks that may have overlappingtask information (e.g., a same shelf location, a same aisle identifier,etc.).

In some embodiments, the indication that a person completed a task(e.g., a task received at step 3402) may be based on an analysis of theat least one image. For example, the at least one processor may analyzeat least one image to determine whether a threshold amount ofinformation is derivable from the at least one image. Continuing thisexample, the at least one processor may determine whether at least oneshelf, at least one product, at least one product placement, at leastone product orientation; at least one sign placement, at least one signorientation, and/or other product information may be derived from the atleast one image (e.g., according to an image analysis). Additionally oralternatively, the at least one processor may determine if the at leastone image has an orientation, focus, resolution, brightness, contrast,and/or the like, which may be conditions for further image analysis(e.g., at step 3406). In some examples, the at least one processor maydetermine that the person is leaving an area of the retail storecorresponding to the task (e.g., according to image analysis to identifya representation of the person in an image and then further imageanalysis revealing an absence of the person in one or more subsequentlycaptured images, and the indication that the person completed the taskreceived at step 3402 may be based on the determination that the personis leaving the area. Alternatively or additionally, location data from adevice associated with the person indicate whether the person has leftthe retail store.

In some examples, a machine learning model may be trained using trainingexamples to determine whether tasks are complete from images, and thetrained machine learning model may be used to analyze the at least oneimage and determine whether the person completed the task. For example,a training example may include an image showing a completed task, apartially complete task, an uncomplete task, etc. By way of furtherexample, a training example may include a sample image and an indicationof a particular task, together with a label indicating whether theparticular task is completed.

In some examples, a convolution of at least part of the at least oneimage may be calculated, and the indication that the person completedthe task received at step 3402 may be based on the calculatedconvolution. For example, in response to a first value of theconvolution, the indication may be provided, and in response to a secondvalue of the convolution, providing the indication may be withheldand/or forwent. Alternatively, the indication that a person completed atask received at step 3402 may be based on receiving at least one image(e.g., on receiving at least one image from a user device, not based onimage analysis).

At step 3404, the at least one processor may receive at least one imageof the at least one shelf. For example, the at least one image may beread from memory, may be received from an external device, may becaptured using an image sensor, and so forth. In some embodiments, theat least one image of the at least one shelf may be captured using animage sensor after the completion of a task (e.g., the task indicated atstep 3402). In some embodiments, the at least one image may be at leastone image captured by at least one of a person (e.g., a person using adevice having an image sensor, the person of step 3402, etc.), by animage sensor mounted to a shelf (e.g., an image sensor mounted on oneshelf such that it faces another shelf), by an image capturing robot, byan indoor drone, and so forth.

At step 3406, the at least one processor may analyze at least one image(such as the at least one image received at step 3404) to determine atleast one property associated with performing a task (e.g., the taskindicated at step 3402). In some embodiments, the at least one propertyassociated with performing the task may include a quality indication ofat least one aspect of performing the task. In some embodiments,analyzing the at least one image may include one or more of the steps ofprocess 3500, discussed further below, and/or aspects discussed above.For example, analyzing at least one image to determine the at least oneproperty associated with performing the task may comprise identifying atleast one visual indicator in the at least one image. In someembodiments, at step 3406, process 3400 may include analyzing the atleast one image received at step 3404 to determine that the person(e.g., the person that completed the task) performed a positive actioncorresponding to the at least one shelf in the retail store. Thepositive action may or may not be included in the task. For example, thetask may have included adjusting a facing of a first product on the atleast one shelf, but the person may also have performed a positiveaction of pulling forward a second product and thereby improved facingsat the at least one shelf. In some embodiments, the at least oneprocessor may determine or change the quality indication of the at leastone aspect of performing the task based on a received input (e.g., aninput received from sensors configured to be positioned between a shelfand product placed on the shelf, from a pressure sensor, from a touchsensor, from a weight sensor, from a light sensor, etc.). In someembodiments, the at least one property associated with performing thetask may be based on a calculated at least one convolution of at leastpart of the at least one image received at step 3404 (discussed furtherbelow). For example, in response to a first value of a convolution ofthe at least part of the at least one image, a first value of a propertyassociated with performing the task may be determined. Also, in responseto a second value of the convolution of the at least part of the atleast one image, a second value of the property associated withperforming the task may be determined. In some embodiments, the secondvalue of the property may differ from the first value of the property.In some embodiments, the at least one processor may differentiatebetween image aspects relevant and irrelevant to a task. For example,the at least one processor may identify at least one product and/orshelf relevant to a task, and at least one product and/or shelfirrelevant to a task. In some embodiments, the at least one processormay ignore the irrelevant image aspects and perform particular imageprocessing steps on the relevant image aspects, consistent withdisclosed embodiments.

In some embodiments, a machine learning model may be trained usingtraining examples to determine properties associated with performance oftasks from images, and step 3406 may use the trained machine learningmodel to analyze the at least one image and determine the propertyassociated with performing a task. For example, a training example mayinclude an image showing a completed task, a partially complete task, anuncomplete task, a task property (e.g., a quality indication), etc. Byway of further example, a training example may include a sample imagerelated to a sample task together with a label indicating a propertyassociated with performing the sample task. In some embodiments, the atleast one image may be analyzed to determine placement of products on atleast part of a shelf, and the at least one property associated withperforming the task may be determined based on the determined placementof products on the at least part of a shelf. In some embodiments, the atleast one image may be analyzed to determine planogram complianceindicators (or other product facing indicators), and the at least oneproperty associated with performing the task may be determined based onthe determined planogram compliance indicators.

In some embodiments, process 3400 may also include analyzing the atleast one image (for example, the at least one image received at step3404) to determine a property of the person. A property of the personmay include a historical degree of task performance quality associatedwith the person (e.g., a quality level of task-related images from theperson, a quality level associated with historical tasks performed bythe person, an experience level of the person, and so on). For example,a face recognition algorithm may be used to analyze the at least oneimage and identify the person, and the determined identity may be usedto determine the property of the person. For example, the determinedidentity may be used to access a database including the property of theperson. In another example, the determined identity may be compared withan expected identity of the person, to validate that the task isperformed by the user. In some embodiments, process 3400 may includeanalyzing the at least one image (for example, the at least one imagereceived at step 3404) to determine at least one additional availabletask corresponding to the at least one shelf in the retail store andproviding an indication of the additional available task to the person.For example, the at least one processor may provide an indication of theadditional available task, which may include a first indication of afirst reward for performing the task, and a second indication of asecond reward for performing the task and the additional available task.For example, the analysis of the image may indicate an issue related tothe at least one shelf (such as a misplaced product on the at least oneshelf, a need to restock the at least one shelf, a need to correctfacings at the at least one shelf, a need to remove products from the atleast one shelf, a need to remove label from the at least one shelf, aneed to improve planogram compliance of the at least one shelf, and soforth), and the at least one additional available task may include aremedy to the issue. In some embodiments, process 3400 may includedetermining an impact of the performed task. For example, the at leastone processor may determine an impact on sales, an impact on futuretasks corresponding to the at least one shelf in the retail store, animpact on engagement of customers, an impact on product facingcompliance, an impact on planogram compliance, an impact determined byanalyzing the at least one image, etc.

In some embodiments, the at least one processor may analyze the at leastone image (for example, the at least one image received at step 3404) todetermine at least one aspect lacking in the performance of the task.For example, the at least one processor may determine, based on imageanalysis, that one or more products on a shelf are not part of aplanogram, that a price number is obscured from view, that less than atasked number of products have been restocked, etc. In some embodiments,process 3400 may include providing an indication of the at least oneaspect to the person (e.g., to a device associated with the person). Insome embodiments, process 3400 may include receiving at least oneadditional image of the at least one shelf, and the at least one imagemay be captured using the image sensor after the indication of the atleast one aspect is provided.

In addition to, or instead of, analyzing images, the at least oneprocessor may also receive other kinds of information. For example,process 3400 may include receiving input from at least one pressuresensor positioned on the at least one shelf. In some embodiments, the atleast one processor may use the input to confirm whether informationdetermined from image analysis is correct. Additionally oralternatively, process 3400 may include using received information todetermine the at least one property associated with performing the task(e.g., the task indicated at step 3402). For example, a machine learningmodel may be trained using training examples to determine properties ofperformance of tasks from one or more kinds of information, and thetrained machine learning model may be used to analyze the information todetermine the at least one property associated with performing the task.Examples of the kinds of information that process 3400 may receive asinput and/or analyze (including information that may be used by atrained machine learning model) include information captured usingpressure sensors, information captured using touch sensors, informationcaptured using weight sensors, light sensors, information captured usingsensors configured to be positioned between a shelf and product placedon the shelf, etc.) related to a sample task, together with a labelindicating a property associated with performing the sample task. Forexample, the at least one processor may compare a pressure detected bythe at least one pressure sensor and compare the pressure to anestimated pressure based on a number of products detected in an image.Additionally, or alternatively, process 3400 may include receiving aninput from at least one touch sensor positioned on the at least oneshelf. Additionally, or alternatively, process 3400 may includereceiving an input from at least one weight sensor positioned on the atleast one shelf. Additionally, or alternatively, process 3400 mayinclude receiving an input from at least one light sensor positioned onthe at least one shelf. For example, the at least one processor maycompare an amount of light detected by the at least one light sensor andcompare the amount of light to an estimated amount of light based on anumber of products detected in an image (e.g., products which may coverthe at least one light sensor to reduce the amount of light detected).

At step 3408, the at least one processor may use the at least oneproperty (e.g., determined at step 3406) to determine a reward forperforming a task. In some embodiments, the reward may include amonetary reward, and the determination of the reward may include adetermination of an amount associated with the monetary reward. A rewardmay include money, a gift card, a promotional benefit (e.g., a percentdiscount off a purchase at the retail store), a coupon, a bonus, creditpoints, increase to a reputation score, and/or any other form ofcompensation. In some embodiments, the at least one processor may alsocause a reward information to be delivered to a person and/or device(e.g., a person and/or device associated with completing a task). Forexample, information configured to cause the reward to be transmitted toan external system, stored in memory, provided to a user (e.g., at auser device), the like, and/or any combination thereof. By way offurther example, the at least one processor (or other device) maytransmit information to cause an electronic transfer of funds from afirst account not associated with the person to a second accountassociated with the person. In some embodiments, a notification of thetransfer may be transmitted to a device associated with the user (whichmay or may not be a device associated with completion of the task).

In some embodiments, the determination of the reward may be based on thequality indication (described above with respect to step 3406).Additionally or alternatively, process 3400 may include basing thedetermination of the reward on a property of the person (describedabove). Additionally or alternatively, process 3400 may include basingthe determination of the reward on a positive action (discussed above).Additionally or alternatively, process 3400 may include basing thedetermination of the reward on the impact (discussed above). In someembodiments, process 3400 may include analyzing at least one additionalimage to determine the reward for performing the task (e.g., anadditional image received in response to notifying a person about anaspect lacking in the performance of an initial task). For example, anindication of the at least one aspect may include an indication of afirst reward for performing the task without addressing the at least oneaspect, and an indication of a second reward for performing the task andaddressing the at least one aspect. Additionally or alternatively,process 3400 may include basing the determination of the at least oneproperty on an analysis of a received input (e.g., an input receivedfrom a pressure sensor, a touch sensor, a weight sensor, a light sensor,etc.). In some embodiments, a reward may be determined in response to anindication that a user has completed a threshold number of tasks. Forexample, after an indication is received that a person has completedthree tasks in the retail store, a discount coupon code may betransmitted to a device associated with the person.

FIG. 35 depicts a flowchart of exemplary process 3500 for performingimage analysis. In some embodiments, process 3500 may be used to helpdetermine task performance information and/or a reward for performanceof a task. For example, all or part of process 3500 may be performed aspart of step 3406 of process 3400. For purposes of illustration, in thefollowing description, reference is made to certain components of system100. For example, any combination of steps of process 3500 may beperformed by at least one processor of a device such a handheld device(e.g., a smartphone, a tablet, a mobile station, a personal digitalassistant, a laptop, and more), a wearable device (e.g., smart glasses,a smartwatch, a clip-on camera), and/or server. Examples of such devices(e.g., capturing device 125, server 135) are described above. It will beappreciated, however, that other implementations are possible and thatother components may be utilized to implement the exemplary process3500. It will also be readily appreciated that the illustrated methodcan be altered to modify the order of steps, repeat steps, delete steps,or further include additional steps. While certain aspects are describedin the context of a retail store, it is appreciated that any or all ofthe steps described could be implemented in other environments, such asa warehouse, fulfillment center, stockpile, or other place where a usermay attempt to complete a task related to a product.

At step 3502, at least one processor (e.g., a processor of an imageprocessing unit 130) may determine image context. Image context mayinclude a location (e.g., an aisle within a retail store, a coordinateposition of a user device, etc.), a task (described above), productinformation (described above), a directionality (e.g., a direction animage sensor was facing when an image was captured), or any otherinformation associated with a task or image. Determining image contextmay include receiving image context from a user device (e.g., a deviceassociated with a person associated with performing a task) and/orreceiving image context from stored information (e.g., image data storedat database 140, which may have been received from a user device).

At step 3504, the at least one processor may access reference imagedata. In some embodiments, reference image data may include an image(e.g., an image of a previous shelf view), a number of objects, a layoutof objects, a shape outline, a color value (e.g., a red-green-blue, orRGB value, a hue value, a saturation value, a light value, etc.), acolor distribution, a color gradient, a contrast level, a logo, aletter, a word, a phrase, or any other criterion to which image data maybe compared. hi some embodiments, reference data may be stored at adatabase (e.g., database 140). In some embodiments, reference image datamay be accessed based on image context. For example, if image contextincludes a task (e.g., changing a price display), image data associatedwith that task may be accessed (e.g., an image of a changed pricedisplay). In some embodiments, accessing reference image data mayinclude generating reference image data. For example, if image contextincludes a task associated with restocking four rows of a product on ashelf and arranging the front product in each row to form a planogram,the at least one processor may access an image of the product andmultiply the image four times, to create an artificial form of what theplanogram may look like in a real-world environment. In someembodiments, reference image data may be part of all of a product model(discussed above).

At step 3506, the at least one processor may extract image features,which may be extracted from a received image (e.g., received from adevice associated with a person associated with performing a task). Forexample, the at least one processor may extract a shape, a color value,an object, a character, a word, or any other visual indicator related toa product from at least one image. The visual indicator may comprise atleast one of: a color, a brightness, a character, or a shape.Additionally or alternatively, the at least one processor may change acolor value of at least one pixel, change a color value of an image(e.g., contrast, brightness, etc.), rotate an image, crop an image,expand an image, shrink an image, or perform any other imagemanipulation, which may make the image data more suitable for comparison(e.g., at step 3508). For example, at least one convolution of at leastpart of at least one image may be calculated (e.g., by image processingunit 130). For example, the at least one processor may apply a matrix toimage data to alter pixel values (e.g., increase a brightness of anypixel below a threshold brightness level), which may improve the imagedata for comparison.

At step 3508, the at least one processor may compare image features(e.g., extracted at step 3506) to reference image data (e.g., accessedat step 3504). For example, the at least one processor may compare theat least one visual indicator (e.g., extracted at step 3506) to areference visual indicator. Comparing image features to reference imagedata may comprise comparing two or more shapes, two or more colorvalues, two or more color value distributions, etc. In some embodiments,the at least one processor may compare an image (e.g., with or withoutany feature extraction), which may be received from a user device, toreference image data (e.g., another image).

At step 3510, the at least one processor may determine a degree ofsimilarity, which may be a degree of similarity between at least oneimage feature and reference image data (e.g., based on the comparison atstep 3508). For example, the at least one processor may determine adegree of similarity between an identified at least one visual indicatorand at least one reference visual indicator (e.g., a percent match). Insome embodiments, a quality indication may be based on the degree ofsimilarity. For example, a quality level, which may influence a reward,may increase for (e.g., be correlated with) higher degrees ofsimilarity. By way of example and without limitation, a planogramextracted from a received image that is a 95% match with reference imagedata may be associated with a better reward (e.g., a higher monetaryamount) than a planogram extracted from a received image that is a 70%match with reference image data. In some embodiments, a determination ofa percent match below a threshold level may be associated with noreward, and may cause an indication (e.g., an indication of no reward,an indication of a lacking aspect, etc.) to be transmitted to a device(e.g., a user device associated with a person associated with performinga task).

As discussed previously, in embodiments of the present disclosure,server 135 may receive image data acquired by crowd sourcing. Forexample, in some situations, cameras placed throughout a location maynot be positioned with a proper angle to view an area. Further, a marketresearch entity or a product supplier may desire information of productplacement within a retail location, but lack access to images capturedby cameras that are operated by a retailer. Further still, a retailermay find it easier and more cost effective to forego at least somecameras and other monitoring systems, and instead rely upon crowdsourced images. Crowd sourcing may also provide dynamic monitoringcapability. For example, a retailer may be unable to anticipate whatmonitoring capabilities are needed and where, or may routinely changeproduct placement and wish to avoid reconfiguring installed monitoringhardware. Rather than placing excess cameras throughout a location andlater discovering that some cameras are not needed or have low userates, or acquiring multiple robotic monitoring units, a retailer mayprefer the flexibility of tasking humans to perform monitoring tasks. Insome examples, a supplier, a retailer or an operator of a retail storemay find it easier and more cost effective to perform some tasks in aretail store through crowd sourcing. Some non-limiting examples of suchtasks may include restocking, removing products from a shelf or adisplay, adding products to a shelf or a display, rearranging productson a shelf or a display, placing a label, removing a label, modifying alabel, changing price of a product, placing promotional material,removing promotional material, rearranging promotional material,promoting a product, communicating with employees or other storeassociates of the retail store, capturing images and/or videos from theretail store, cleaning the retail store, scanning barcodes (or othervisual codes) in the retail store, and so forth.

FIG. 36 is a schematic illustration of a communications networksupporting multiple users, consistent with the present disclosure. Thecommunications network illustrated in FIG. 36 may be used to supportcrowd sourced image and data capture of places within a retail store. Asillustrated in FIG. 36, server 135 of image processing unit 130 may beconnected to communications network 150. Further, individual userdevices 3602B, 3604B, and 3606B associated, respectively, with users3602A, 3604A, and 3606A, may communicate via communications network 150with server 135. Server 135 may receive assignments (e.g., from aretailer, from a supplier, from an operator of a retail store, etc.),select an assignment, and transmit the assignment to a user device viacommunications network 150. Users 3602A, 3604A, and 3606A may perform areceived assignment using respective user devices, and user devices3602B, 3604B, and 3606B may then relay data collected while performingan assignment to server 135 via communications network 150. For example,network 150 may include or be part of the Internet, a Local AreaNetwork, wireless network (e.g., a Wi-Fi/302.11 network, a 4G or 5Gcellular network), or other suitable connections.

As shown in FIG. 36, user devices 3602B, 3604B, and 3606B may beassociated with differing device capabilities. For example, user device3602B may be a phone with a 5G connection, 256 GB of memory, and a 12 MPcamera. Other user devices may have different capabilities. Forinstance, in comparison with user device 3602B, user device 3606B mayhave a slower connection (4G rather than 5G), less memory (64 GB ratherthan 256 GB), and a lower resolution camera (5 MP rather than 12 MP).

In some scenarios, assignments processed by server 135 may make use ofor necessitate differing device capabilities. For instance, anassignment may involve uploading a large, high resolution video, inwhich case, a user device with high connection speed and cameraresolution may be necessary. A different assignment may require an imagewith only a low resolution, such that a device with lesser capabilitiesmay be sufficient. Accordingly, server 135 may determine devicerequirements based on assignment needs. Server 135 may also determinedevice requirements based on capabilities of server 135. For example,server 135 may be configured to perform image enhancement functions,such as improving resolution in order to perform optical characterrecognition. Thus, for example, server 135 may determine that deviceswith a lower resolution camera are sufficient, as a low resolution imagemay be enhanced by server 135 and still satisfy an assignment.

Server 135 may also determine environmental conditions of user devicesin addition to hardware capabilities. For example, if a user hasselected to connect a phone to a store's WiFi network, server 135 maydetect a connection to WiFi and provide assignments to the phone thatrequire large amounts of data transfer, while refraining from sendinglarge data transfer assignments to user devices only connected to acellular network.

As discussed above, users may receive incentives in exchange forperforming an assignment. Server 135 may calculate an incentive valuebased on, for example, savings that a retailer may realize by opting forcrowd sourced monitoring rather than installing monitors or usingrobotic monitors. The incentive may alternatively or additionally takeinto account a likelihood of a user performing an assignment, theassignment's value to a market analysis, and/or the difficulty of theassignment. Server 135 may provide information to user devices regardingthe assignment and incentive, and the user devices may display theassignment as an offer in a graphical user interface for the user toaccept or decline.

FIG. 37 is a schematic illustration of a user interface, consistent withthe present disclosure. User interface 3702 may be displayed on a userdevice 3602B, such as a phone, to a user, such as user 3602A. Userinterface 3702 may provide a description of the task and an incentive.For example, user interface 3702 may offer a user a $1 coupon inexchange for taking a picture, which user device 3602B may send toserver 135. The incentive may be related to the assignment, as shown(e.g., the coupon shown is for detergent in exchange for a picture ofthe laundry aisle), or may be unrelated. User interface 3702 may alsoprovide a user an option to accept the offer or decline the offer. Insome embodiments, user interface 3702 may display when a user opens anapp. User interface 3702 may also display as a text message, pop upmessage, and the like. In some embodiments, server 135 may send a set ofassignments to user device 3602B, which user device 3602 may displaywhen the user is capable of performing the assignment or when the useris likely to accept the assignment (such as being within a certainradius of a task area, or logging into a WiFi network). Further, userdevice 3702 may report conditions to server 135 in real time, and server135 may cause a user interface to display based on the real timereporting. The assignment shown in FIG. 37 is an example, andassignments may involve multiple tasks and/or more specific directions(e.g., take a picture of a specific shelf and/or a specific product).

FIG. 38 provides a flowchart of an exemplary method for providingassignments to users, consistent with the present disclosure. Server 135may include at least one processor programmed to perform the steps ofprocess 3800 illustrated in FIG. 38.

At step 3802, server 135 may receive an indication of at least oneparameter of a mobile computing device associated with a user. A mobilecomputing device may include, for example, a cellular phone (e.g., asmartphone with a built-in camera), a tablet, a smartwatch, a wearablecomputing device, an augmented reality system, a virtual reality system,etc. A mobile computing device may push at least one indication ofparameters or status, for instance periodically, or server 135 mayrequest indications. Parameters may be related to an image sensor,computing power, and/or memory capacity of the mobile computing device.

For example, the at least one parameter of the mobile computing devicemay be at least one parameter of an image sensor of the mobile computingdevice associated with the user. The at least one parameter of the imagesensor may comprise at least one of a maximal pixel resolution, maximalframe rate, type of sensor, and zoom capability. Other non-limitingexamples of such parameters of the image sensor may include size, pixelsize, color parameters, range of focal lengths, maximal field of view,sensitivity to light, any indicator of quality of the image sensor, andso forth. Some non-limiting examples of types of sensors may includedigital, non-digital, Charge-Coupled Device (CCD), Complementary MetalOxide Semiconductor (CMOS), 2D, 3D, mono, stereo, depth, active stereo,time-of-flight, Light Detection and Ranging (LIDAR), and so forth. Insome embodiments, the mobile computing device may indicate a camera typeor mobile computing device type, and server 135 may look up cameracapabilities based on the camera type or mobile computing device type.

Further, the at least one parameter of the mobile computing device maybe at least one parameter of a computing power of the mobile computingdevice associated with the user. The at least one parameter of thecomputing power may comprise at least one of an existence of a GPU, anumber of computing cores, a type of processor, and a number ofinstructions per time unit. The mobile computing device may indicate adevice type, and server 135 may look up computing parameters in adatabase for instance. In some embodiments, server 135 may determine acomputing power by providing a computational task to the mobilecomputing device. For example, server 135 may provide a nonce task, suchas adding numbers repeatedly until a sum exceeds a threshold, anddetermine the number of operations the mobile computational deviceperforms in a time unit. This may have an advantage of providingreal-time computing power, reflecting other tasks simultaneously beingperformed by the mobile computing device, rather than a notionalcomputing power based on device specifications.

Further still, the at least one parameter of the mobile computing devicemay include at least one parameter of a memory of the mobile computingdevice associated with the user. For example, the at least one parameterof a memory may comprise at least one of a memory size (such as totalsize of memory, size of free available memory, etc.), type of memory,and speed of memory. The mobile computing device may indicate a devicetype, and server 135 may look up memory parameters in a database storedas part of server 125 or accessible to server 135 over a wired and/orwireless network. In some embodiments, server 135 may determine memoryparameters by providing a memory task to the mobile computing device,such as sending a file to the mobile computing device and timing howlong the mobile computing device requires to store and recall the file.

At step 3804, server 135 may access a plurality of availableassignments, each assignment of the plurality of available assignmentsinvolving use of at least one mobile computing device. Retailers,operators of retail stores, market researchers, and/or suppliers mayprovide possible assignments to server 135, and server 135 may trackwhich of the possible assignments have been completed, which of thepossible assignments are assigned to users, and which have not yet beencompleted and are not assigned to users and are thus available. Possibleassignments may involve a number of iterations. For instance, a marketresearcher may request 10 pictures of a product in a grocery store(e.g., in order to obtain images of a desired quality and/orperspective). Server 135 may also determine if an assignment isavailable based on a completion criteria. For example, an assignment mayinvolve capturing an image of a price of an item on a shelf. A firstmobile computing device user may attempt to complete the assignment, butthe image may be of too low quality for server 135 to determine theprice. Server 135 may continue marking the assignment as being availableuntil an image is captured with sufficient quality for server 135 toextract a price (e.g., via optical character recognition techniques).Server 135 may then remove the assignment from the plurality ofavailable assignments.

Other assignments may use other aspects of at least one mobile computingdevice, such as recording audio in a retail location, recording a videoof a customer interaction, real-time video and/or audio analysiscapabilities, or recording WiFi or cellular signal strength. Anassignment may involve the use of multiple mobile computing devices,such as using a microphone of one device and a camera of another device,either in sequence or simultaneously.

At step 3806, server 135 may begin selecting a subset of the pluralityof available assignments, based on the at least one parameter of themobile computing device associated with the user. Server 135 may revieweach of the available assignments, and select a subset of the pluralityof available assignments.

For instance, server 135 may determine whether a particular assignmentof the plurality of available assignments is compatible with the atleast one parameter of the mobile computing device associated with theuser. Compatibility may indicate that the mobile computing device iscapable of performing the selected assignment. For instance, the mobilecomputing device computational power may exceed a computing powerrequired to perform the assignment.

In response to a determination that the particular assignment of theplurality of available assignments is compatible with the at least oneparameter of the mobile computing device associated with the user,server 135 may include the particular assignment in the selected subset.In other words, step 3806 is YES, and server 135 proceeds to include theassignment in the selected subset at step 3808.

However, in response to a determination that the particular assignmentof the plurality of available assignments is incompatible with the atleast one parameter of the mobile computing device associated with theuser, server 135 may forgo including the particular assignment in theselected subset. Thus, step 3806 is NO, and server 135 proceeds toinclude the assignment in the selected subset at step 3810.

In some embodiments, server 135 may also determine compatibility basedon historical assignments provided to a mobile computing device. Forexample, server 135 may, at step 3806, identify an historic assignmentmatching the particular assignment. The historic assignment and theparticular assignment may have a matching category, such as imagecapture, or a common mobile computing device requirement, such as cameraresolution. Server 135 may also identify a particular mobile computingdevice matching the mobile computing device associated with the user.The particular mobile computing device may be the same type of mobilecomputing device, such as a phone model, or may be the same phoneitself. Further, server 135 may access a success indicationcorresponding to a historic allocation of the assignment to theparticular mobile computing device. Server 135 may then use the successindication to determine whether the particular assignment of theplurality of available assignments is compatible with the at least oneparameter of the mobile computing device associated with the user.Accordingly, server 135 may avoid including assignments in the subsetthat are similar to assignments previously excluded from subsets for aparticular device, or that are similar to assignments that theparticular device has previously failed to accomplish. In some examples,server 135 may analyze one or more images captured by the mobilecomputing device of the user (for example, as part of historicalassignments provided to the mobile computing device), and may determinewhether to include a particular assignment in the subset of theplurality of assignments based on a result of the analysis. For example,a convolution of at least part of an image captured by the mobilecomputing device of the user may be calculated, in response to a firstvalue of the calculated convolution, the particular assignment may beincluded in the subset of the plurality of assignments, and in responseto a second value of the calculated convolution, the particularassignment may be excluded from the subset of the plurality ofassignments.

Server 135 may proceed to step 3812 and determine whether the subset iscomplete. Step 3812 may therefore include determining if there are anyremaining available assignments that have not been compared to the atleast one parameter. Step 3812 may also or alternatively includecomparing a number of assignments in the subset to a threshold, suchthat the subset does not exceed a predetermined size. In one example, ifthere are remaining available assignments or the subset is smaller thana threshold size, the subset is not complete and step 3812 is NO, andserver 132 may return to step 3806 and compare an additional assignmentwith the at least one parameter. Otherwise, if the subset is complete(in this example, all assignments have been compared to the at least oneparameter and/or the subset has reached the predetermined size), step3812 is YES, and server 132 proceeds to step 3814. Step 3812 may useother rules for determining the completeness of the subset.

Server 135 may then proceed to step 3814, and offer the selected subsetof the plurality of available assignments to the user. In some cases,the subset may include one assignment, as discussed herein in an exampleembodiment, or the subset may include a plurality of assignments thatmay, for instance, have a common assignment type (i.e., taking apicture, being located in a particular store, and the like). The mobilecomputing device associated with the user may display the offer, such asby a graphical user interface as shown in FIG. 37 and may includeoffering the user a coupon in exchange for accepting at least oneassignment in the selected subset. Server 135 may receive an indicationof accepting or declining the assignment from the mobile computingdevice.

In some scenarios, even if a mobile computing device is compatible withan assignment, and the user has accepted the assignment, the user and/orthe mobile computing device may still fail to perform the assignment.

For instance, if the user accepts the assignment, server 135 may provideadditional data required to perform the assignment, such as furtherinstructions for display to the user, or instructions to be performed bythe mobile computing device. Thus, a particular assignment of theselected subset may include a precondition, a first instruction toperform the particular assignment if the precondition is satisfied, anda second instruction to forego performing the particular assignment ifthe precondition is not satisfied. To further illustrate, an assignmentmay involve taking a picture of a product in a store. However, for thepicture to be usable by a requesting party, the picture may need to bebright enough to analyze, that an object size in the image issufficient, etc., and server 135 may therefore include a precondition ofat least one of a brightness level or a proximity to an object in animage. The mobile computing device may, in some embodiments, performinstructions to test the precondition, such as measuring a brightnesslevel (for example, with a light sensor, by analyzing the image, etc.),measuring a distance from a target object (for example, using aproximity sensor, by analyzing the image, etc.), and so forth. Thus,even though a mobile computing device may be compatible with theassignment, compatibility alone may be insufficient to perform theassignment. Further, determining if an assignment is compatible with themobile computing device at step 3806 may include determining if themobile computing device is able to perform the precondition, in additionto the assignment.

In some embodiments, a user may be provided with a guide or instructionsto perform the assignment. For instance, one or more assignments of theselected subset may further include respective remediation instructions,the remediation instructions instructing the user to change a positionof the mobile computing device associated with the user in relation toan object in response to the precondition not being satisfied. Forexample, a user of a mobile computing device may be presented with aninstruction to move the mobile computing device closer to an item inorder to take a clearer picture.

Server 135 may also provide a quality requirement, which may be used bya mobile computing device to determine if the precondition is satisfied.For example a particular assignment of the selected subset may include aquality requirement. The mobile computing device may compare captureddata to the quality requirement. Server 135 may receive, from the mobilecomputing device associated with the user, a failure indication inresponse to the mobile computing device associated with the userperforming the particular assignment of the selected subset and failingone or more criteria of the quality requirement. In one example, themobile computing device associated with the user may have a cameracompatible to the particular assignment, but due to unforeseen conditionof the mobile computing device associated with the user (such as ascratched lens, an unclean lens, etc.), the quality of captured imagesmay be insufficient. In this example, server 135 may also provide aremediation instruction to the user to clean the lens. In anotherexample, the mobile computing device associated with the user may have acamera compatible to the particular assignment, but due to executionproblems (such as trembles, incorrect position or aim, etc.), thequality of captured images may be insufficient. For example, thecaptured image may be analyzed to determine that the sharpness of theimage is sufficient, that a size of a particular object (such as aproduct, a label, etc.) is sufficient, and so forth. In case theanalysis determines an insufficiency, server 135 may also provideappropriate remediation instruction to the user.

A user, however, may not wish to perform remediation instructions. Forexample, a user may become frustrated with repeated or difficultinstructions, and may cease performing assignments. Further, aprecondition may fail due to circumstances that cannot be remedied by auser, such as poor or slow connection, store conditions, weatherconditions, and the like. Accordingly, in some embodiments, server 135may provide an instruction to the mobile computing device associatedwith the user to complete a second assignment of the selected subset inresponse to not satisfying a precondition of a first assignment.

A mobile computing device may have sufficient hardware to perform anassignment, but may not have sufficient unallocated capacity to performthe assignment. For example, as discussed previously, a mobile computingdevice may have 64 GB of memory installed, but only 10 MB of memoryfree. Accordingly, an assignment that requires 500 MB may beincompatible with the mobile computing device. However, server 135 maynot have information of the free memory when determining compatibilityat step 3806. Alternatively, a mobile computing device may have 500 MBof memory free when server 135 determines assignment compatibility, butmay only have 300 MB of memory free when the mobile computing deviceactually performs an assignment. Thus, server 135 may receive anindication from the mobile computing device associated with the userthat the mobile computing device associated with the user lacks at leastone resource to perform at least one assignment of the selected subset.Server 135 may change a status of the at least one assignment fromunavailable to available, provide a second assignment of the subset tothe user device, remove similar assignments from the subset of theplurality of available assignments, and/or may store an indication thatthe particular mobile computing device is incompatible with assignmentsof the same type.

Further, a user may not grant adequate permissions for the mobilecomputing device to perform an assignment. In such embodiments, server135 may receive an indication from the mobile computing deviceassociated with the user that the mobile computing device associatedwith the user does not have permission to perform an assignment of theselected subset. As discussed previously, server 135 may change a statusof the at least one assignment from unavailable to available, provide asecond assignment of the subset to the user device, remove similarassignments from the subset of the plurality of available assignments,and/or may store an indication that the particular mobile computingdevice is incompatible with assignments of the same type.

In some cases, a user may decline an assignment due to time constraints,disinterest, or an inadequate incentive. In response, server 135 mayattempt to provide the user an alternate assignment, and may tailorassignments based on user preferences indicated by the user declining anassignment. For instance, server 135 may receive a response indicationthat the user has declined the offered selected subset of assignments.Server 135 may also determine a category of the selected subset ofassignments.

For example, a user may decline a subset of assignments requiring theuser to record conversations with customer service representatives, andthe category of the selected subset may be “audio recordingassignments.” Server 135 may proceed to select a new subset of theplurality of available assignments, and the new subset may excludeassignments of the determined category. For instance, server 135 mayperform process 3800, and eliminate assignments of the determinedcategory at step 3806 as incompatible with the mobile computing deviceand/or user specifications. Server 135 may then offer the new subset tothe user. In this manner, server 135 may avoid annoyance and missedproductivity of a particular user.

As discussed previously, in embodiments of the present disclosure,server 135 may receive image data acquired by crowd sourcing throughassignments distributed to users. In addition to managing crowd sourcingassignments within retail stores, disclosed systems and methods may alsobe adapted to manage assignments that take place external to retailstores, or mix a retail store assignment with an assignment external toretail stores. This may provide benefits of increased efficiency inperforming assignments, increased assignment opportunities for users,and optimization of a user's time through merging assignment types andminimizing down time. For example, if a driver for hire provides someonea ride to a store or to a location near the store, the driver may beable to perform an additional assignment inside the store, which mayprovide an additional benefit to the driver and increase the value ofdriving the person to the store. This may help satisfy assignments for arural store, as well, and make trips to rural or remote locations moreprofitable for a user. As used herein, an assignment in a retail storemay refer to a task wherein a user enters a store, such as to take apicture of a display, purchase an item, or drop off an item, or mayrefer to a task wherein the user visits a store, such as going through adrive-through, takes a picture of the outside of the store, or drops offan item in a receptacle. Some non-limiting examples of such assignmentsmay include entering the retail store, restocking a shelf or a display,removing products from a shelf or a display, adding products to a shelfor a display, rearranging products on a shelf or a display, placing alabel, removing a label, modifying a label, changing price of a product,placing promotional material, removing promotional material, rearrangingpromotional material, promoting a product, communicating with employeesor other store associates of the retail store, capturing images and/orvideos from the retail store, cleaning the retail store, scanningbarcodes (or other visual codes) in the retail store, purchasing anitem, and so forth.

FIG. 39 is a schematic illustration of a communications networksupporting multiple users, consistent with the present disclosure. Thecommunications network illustrated in FIG. 39 may be used to supportassignments for users both internal and external to a retail store. Asillustrated in FIG. 39, server 135 may be connected to communicationsnetwork 150. Further, individual user devices 3902B, 3904B, and 3906Bassociated, respectively, with users 3902A, 3904A, and 3906A, maycommunicate via communications network 150 with server 135. Server 135may receive assignments (e.g., from a retailer, from a supplier, from anoperator of a retail store, etc.) or generate assignments, select anassignment, and transmit the assignment to a user device viacommunications network 150. Users 3902A, 3904A, and 3906A may perform areceived assignment using respective user devices. User devices 3902B,3904B, and 3906B may relay data collected while performing an assignmentto server 135 via communications network 150. For example, network 150may include or be part of the Internet, a Local Area Network, wirelessnetwork (e.g., a Wi-Fi/302.11 network, a 4G or 5G cellular network), orother suitable connections. Some non-limiting examples of individualuser devices 3902B, 3904B, and 3906B may include smartphones,smartwatches, tablets, personal computing devices, personalcommunication devices, wearable computing devices, augmented realitysystems, virtual reality systems, and so forth.

As shown in FIG. 39, users 3902A, 3904A, and 3906A may be associatedwith different vehicles and/or equipment. For example, user 3902A maydrive a truck 3902C, capable of transporting large items. User 3904A maybe associated with a van 3904C which could provide climate-controlledtransportation of large items. User 3906A may be associated with a sedan3906C, which may be suitable for driving in a congested city and parkingin small parking lots. Users may be associated with other equipment orcapabilities, as well, such as a towing capability, dolly, hand truck,high horsepower or torque engines, storage space, and the like. Further,users may also have personal attributes (e.g., ability to lift,knowledge of an area or store, ability to drive with a trailer, oraccess passes to certain areas).

In some scenarios, assignments processed by server 135 may make use ofor necessitate differing personal, equipment, or vehicle capabilities.For instance, an assignment may include an assignment external to aretail store involving delivering a large flower bouquet to anapartment, followed by picking up a bag of mulch from a retail store.This assignment may make use of a user's van for climate control toprotect the bouquet, the user's ability to climb stairs, the user'sability to carry heavy items, and the user's van for moving messy items.Accordingly, server 135 may determine user and/or equipment requirementsbased on assignment needs. Server 135 may also determine devicerequirements based on assignment needs (such as the ability to take highresolution pictures), for example as described above in relation toFIGS. 36, 37 and 38.

Server 135 may also calculate incentives, such as a payments, rewards,bonuses, coupons, credit points, etc., for the user in exchange forperforming an assignment. The incentive may be based on an amount oftime, gas consumption, or assignment difficulty. The incentive mayalternatively or additionally take into account a likelihood of a userperforming an assignment and/or the assignment's value. In somescenarios, a consumer may define assignments, such as picking up drycleaning and bringing it to the consumer's office. A business may alsodefine assignments, such as transporting a part for a car repair fromone shop to another. The consumer or business may submit assignmentrequests to server 135 via a smartphone app or website for instance. Theconsumer or business may also set an offer value associated with theassignment, or server 135 may suggest or set a value based on theassignment details.

Server 135 may provide information to user devices regarding theassignment and incentive, and the user devices may display theassignment as an offer in a graphical user interface for the user toaccept or decline.

FIGS. 40A and 40B are illustrations of selecting retail stores based ona route, consistent with the present disclosure. For example, aselection of an assignment in a retail store may be based on anassignment external to a retail store, such that the assignment externalto a retail store precedes, succeeds, concurs, or overlaps an assignmentinternal to a retail store. For instance, a user may be a driver takinga passenger to a location (i.e., an assignment external to a retailstore), and the user may then be given an assignment to visit a storenear the location (i.e., an assignment internal to a retail store).Alternatively, an assignment external to a retail store may be based onan assignment internal to a retail store. In this case, a user may havean assignment to return a rental item at a home improvement store (i.e.,an assignment internal to a retail store), and receive an assignment todrive someone from the home improvement store to a work site (i.e., anassignment external to a retail store). Further, in some scenarios, aretail store may refer to any business, including stores that sell itemsas well as business locations that provide services. Additionally,assignments may also be related to residences, public spaces, publicbuildings, and commercial locations.

FIG. 40A shows a user 4004 with a planned route to a location 4006within an area 4002. For instance, user 4004 may be a driver taking apassenger to location 4006. Retail stores 4008, 4010, 4012, and 4014 mayalso be within area 4002, and server 135 may have at least one taskcorresponding with each of retail stores 4008, 4010, 4012, and 4014.

An assignment of user 4004 may be time-sensitive. For example, asstated, user 4004 may be driving a passenger to location 4006, and thepassenger may wish to arrive at location 4006 as quickly as possible.Alternatively, user 4004 may be transporting a perishable item, such asgroceries, to location 4006. Thus, the assignment of user 4004 involvesuser 4004 traveling to location 4006, which may be internal or externalto a retail store, may be completed prior to user 4004 performing asubsequent assignment.

Accordingly, server 135 may exclude assignments corresponding to retailstores outside of a threshold radius of location 4006, even if theretail stores are near an initial location of user 4004, or near alocation along the planned or actual route of user 4004, as is the casein FIGS. 40A and 40B. For instance, as shown in FIG. 40B, server 135 mayexclude assignments relating to retail stores 4008, 4010, and 4012 whenidentifying a new assignment for user 4004 to perform after arriving atlocation 4006. Further, server 135 may identify an assignment relatingto retail store 4014 to provide to user 4004, based on retail store 4014being the only retail store with an assignment and within a thresholdradius of location 4006.

In some scenarios, an assignment external to a retail store may not betime-sensitive. For example, user 4004 may be delivering anon-perishable good to location 4006. Server 135 may then provide user4004 with assignments relating to retail stores near an initial locationof user 4004, such as an assignment relating to any of retail stores4008, 4010, and 4012. Further, server 135 may provide multipleassignments related to retail stores in a batch, such as an assignmentrelating to each of retail stores 4008, 4010, and 4012.

In some scenarios, multiple retail stores may be within a thresholdradius of a destination. Server 135 may provide an assignment relatingto the closest retail store to a destination and within a thresholdradius, or may provide multiple assignments within the threshold radiusand allow user 4004 to select at least one of the multiple assignments.Additionally, server 135 may also provide assignments relating to retailstores within a threshold distance of the route of user 4004 to location4006. For example, server 135 may provide an assignment relating to aretail store that requires a detour less than a threshold, for exampleand without limitation, detours requiring less than 10 minutes, lessthan 2 miles, etc.

FIG. 41 provides a flowchart of an exemplary method for selectingavailable assignments, consistent with the present disclosure. Server135 may include at least one processor programmed to perform the stepsof process 4100 illustrated in FIG. 41.

At step 4102, server 135 may receive an indication of an externalassignment, wherein the external assignment may be associated with oneor more locations external to a plurality of retail stores. Theindication of an external assignment may be provided by a mobilecomputing device of a user, including, for example, a cellular phone(e.g., a smartphone with a built-in camera), a tablet, a smartwatch, apersonal computing device, a personal communication device, a wearablecomputing device, an augmented reality system, a virtual reality system,and so forth. A mobile computing device may push the indication of anexternal assignment, for instance periodically or at initiation of theexternal assignment, or server 135 may request indications. Theindication of an external assignment may also be provided by a separateserver which selects and provides external assignments to users. In someembodiments, server 135 may also provide the external assignment. Inanother example, server 135 may access a database or an external deviceto receive the indication of the external assignment.

Further, the external assignment may be associated with multiplelocations, such as an assignment to visit each chain restaurant in anarea to capture images of the restaurant storefronts for qualityassurance purposes. In some embodiments, the external assignment may beassociated with a retail store that is not among the plurality of retailstores. For instance, a company may operate a plurality of retailstores, and the external assignment may be related to a retail storeoperated by a different company.

The external assignment may include an available assignment offered tothe user. Server 135, or another server, may store a list of externalassignments, and may also record which assignments are available, inprogress, or complete. Server 135 may send an indication, such as viacommunications network 150, to a user device of a user with details ofthe available assignment, or details of multiple assignments, which mayinclude, e.g., location, reward or incentive, task, and/or deadline. Auser may select one or more of the available assignments, and the user'sdevice may send an indication to server 135 or another server that theselected one or more assignments should be updated to have an “inprogress” status.

The external assignment may include an assignment selected for the userbased on one or more preferences of the user. Assignments in retailstores may also be selected based on one or more preferences of theuser. User preferences may be stored in the user's device, and theuser's device may select assignments from a list that match the one ormore preferences. Alternatively, server 135 or another server may storethe user preferences in association with a device or user identifier andselect assignments corresponding to the one or more preferences. A usermay provide indications of preferences to a user interface of a userdevice, and the user device may transmit indications of the preferencesto server 135 via communications network 150. The user's preferences mayindicate that the user prefers tasks that allow the user to stay in avehicle, involve transporting people, involve going through drivethroughs, or involve taking images of building exteriors. A user mayalso indicate a preference for tasks that do not involve entering abuilding, or tasks that involve being in a vehicle with goods ratherthan people. Further, users may prefer assignments that may be completedwith one stop, such as taking a picture of an aisle in a grocery store,rather than multiple stops, such as picking up an item from one locationand transporting it to another location.

The external assignment may also or alternatively be determined based ona type of equipment available to the user, such as a vehicle type of theuser, a type of mobile computing device used by the user, capabilitiesof the mobile computing device used by the user. Assignments in retailstores may also be determined based on a type of equipment available tothe user. For example, as discussed previously, a user may have packagehandling equipment such as dollies or hand carts. Equipment availabilitymay be stored in the user's device, and the user's device may selectassignments from a list that match the equipment availability. Server135 may also determine equipment availability. For example, a user mayprovide an indication, via a user interface of a user device, includinga vehicle type or VIN, and server 135 may determine the vehicle'scapabilities by reference to a database of vehicle types or VINs andvehicle characteristics. Server 135 may also determine mobile computingdevice capabilities based on device type, which may be reported by themobile computing device or retrieved by server 135. The mobile computingdevice may indicate a device type, and server 135 may look up memoryparameters in a database for instance stored as part of server 135 oraccessible to server 135 over a wired and/or wireless network. Server135 may also provide test assignments to the user or user device, suchas a test assignment asking a user to take and transmit a picture with auser device, and server 135 may determine image resolution or datatransfer speed. When an assignment is submitted to server 135, theassignment may include a description reflecting the need for certainequipment, and server 135 may match assignments to users based on storedassignment descriptions.

In some embodiments, a user may try to earn money or other rewards byperforming assignments during or between tasks associated with anemployer of the user. The employer may be unassociated with theplurality of retail stores. For example, the user may be a realtor withappointments from a real estate agency employer to show houses topotential buyers throughout a day. The realtor may have extra timebetween appointments, which may allow the realtor to perform anassignment related to a retail store. Similarly, a user may be sent byan employee to an area near a retail store for a meeting or other task,and server 135 may provide the user with an assignment to perform a taskassociated with a retail store in conjunction with or in addition to theemployer's task.

Returning to FIG. 41, at step 4104, server 135 may select a retail storeamong the plurality of retail stores based on the indication of theexternal assignment. The indication of the external assignment mayinclude a transportation assignment associated with a particularlocation, and the selected retail store may be determined based on theparticular location, as discussed previously by reference to FIGS. 40Aand 40B. For instance, the selected retail store may be determined basedon a distance from the particular location to the selected retail store.The distance may be measured in length, such as kilometers, miles, ortime, such as minutes. For example, the selected retail store may bewithin a ten minute drive of the particular location.

Additionally, the indication of the external assignment may also providea planned travel route for a transportation assignment from a firstlocation to a second location. Server 135 may analyze the travel routeand select the retail store based on a measure of a detour to theselected retail store when traveling from the first location to thesecond location. Server 135 may iteratively, for each of the pluralityof retail stores, determine an optimal route and measure an expecteddetour time or distance to each of the plurality of retail stores.Server 135 may select the retail store with the shortest detour. Server135 may include an expected assignment duration in the detourcalculation. For instance, server 135 may, based on expected assignmentduration, select an assignment to take a picture at a retail store witha large detour instead of an assignment to purchase ten items at aretail store with a short detour.

In some embodiments, the external assignment may include being at aparticular location at an estimated time period, and the selected retailstore may be determined based on the estimated time period. Theindication of the external assignment may include an expected time ofarrival, and server 135 may select a retail store among the plurality ofretail stores based on the expected time, such as by comparing theretail store open hours to the expected time of arrival. A retail storeassignment may also be based on a time of day. For instance, a retailstore manager may want pictures of checkout lines at 5:00 PM, and server135 may provide an assignment to take pictures of checkout lines a useranticipated to be near the retail store at 5:00 PM.

In some embodiments, server 135 may select retail stores based ondesired earning rates of users. Users may be unwilling to perform tasksthat do not provide sufficient incentives for an amount of timeinvested. Accordingly, server 135 may receive, from the user, a desiredearning rate. The desired earning rate may be used to determine aminimum total incentive for performing an assignment. For example, auser may desire to earn $10 per hour, thereby indicating that the userdesires to earn $2.50 for a 15 minute assignment. If assignments at aretail store anticipated to require 15 minutes do not provide at least$2.50 in incentives, server 135 may not select the retail store.

Server 135 may further base expected earning rate on travel time.Accordingly, server 135 may determine a travel time to a locationassociated with each of a plurality of alternative retail stores, forinstance as a detour or as an additional destination after completing anexternal assignment. Based on the respective travel times, assignmentcompletion times, and incentives, server 135 may obtain a predictedearning rate for each of the plurality of alternative retail stores.Server 135 may determine the selected retail store based on thepredicted earning rate equaling or exceeding the desired earning rate.In some embodiments, users may desire an earning rate greater than anyavailable assignments offer. Server 135 may provide an indicationcausing user devices associated with these users to display that noavailable assignments exist, along with information of available earningrates of available assignments.

Further, users may desire to deduct travel expenses, such as fuel,tolls, vehicle depreciation, or public transit fees, from expectedearnings to reflect a net earning rate. Accordingly, server 135 mayreceive, from the user, an estimated gas mileage of a vehicle associatedwith the user and a cost per quantity of gas. Server 135 may alsoretrieve estimated gas mileage based on a reported vehicle type or VINfrom a database, and retrieve a cost per quantity of gas from a databaseof average gas prices in an area. Server 135 may estimate, based on theestimated gas mileage and cost per quantity of gas, a total fuel priceassociated with each of the plurality of retail stores. The total fuelprice may be a product of a distance to each of the retail stores, thegas mileage, and the cost per quantity of gas. Server 135 may alsodetermine expected toll fees based on a planned route to a retail store,and vehicle depreciation by reference to a database. Server 135 maydeduct the total fuel price from the predicted earnings and determine apredicted earning rate for an assignment for use in selecting a retailstore in step 4104.

Returning to FIG. 41, at step 4106, server 135 may, based on theselected retail store, select at least one available assignment in theselected retail store. Assignments may be selected randomly from a listof assignments, or selected based on user preferences and/orcapabilities. In some embodiments, such as if retail store selection wasnot performed based on earning rate, server 135 may, at step 4106,determine an amount of time estimated to complete the selected at leastone available assignment, and select assignments based on the amount oftime estimated being less than an amount of available time provided bythe user via a user device to server 135. The selected at least oneavailable assignment may include, for example, purchasing an item at theselected retail store, taking a picture of a display at a store,checking inventory of an item at a store, interacting with storeemployees, or returning an item to a store. Some non-limiting examplesof such assignment may include entering the retail store, restocking ashelf or a display, removing products from a shelf or a display, addingproducts to a shelf or a display, rearranging products on a shelf or adisplay, placing a label, removing a label, modifying a label, changingprice of a product, placing promotional material, removing promotionalmaterial, rearranging promotional material, promoting a product,communicating with employees or other store associates of the retailstore, capturing images and/or videos from the retail store, cleaningthe retail store, scanning barcodes (or other visual codes) in theretail store, purchasing an item, and so forth. In some embodiments, theselected at least one available assignment may include obtaining orleaving an item at a drive through or other drop off of a store.

At step 4108, server 135 may offer the selected at least one availableassignment in the selected retail store to a user. Step 4108 may includesending an indication of an offer to a user device for display,including assignment details such as location and instructions, anamount of time estimated to complete the task (which may include transittime and time at the retail store), and an incentive to perform theselected at least one available assignment. In some cases, a retailstore may be associated with multiple available assignments. Forinstance, a retail store may be associated with a task to take a pictureof a display, check inventory of an item, and purchase a different item.Server 135 may bundle assignments associated with a retail store andoffer the user an additional incentive to complete multiple assignments.Further, server 135 may offer to the user a bundle of the externalassignment and the selected at least one available assignment in theselected retail store. The bundle may include an additional incentive inexchange for performing the external assignment and the at least oneavailable assignment in the selected retail store.

Step 4108 may also include determining a reward corresponding to theselected at least one available assignment based on a property of theexternal assignment, and providing to the user an indication of thedetermined reward. The reward may be a token provided in an app, forinstance, or may be monetary. For example, the reward may be a badgeindicating that the user has performed a task at each retail store of acompany in an area. The reward may be a monetary award based on the usertransporting a passenger to the retail store or delivering an item tothe retail store. The reward may also be based on an external assignmentor selected at least one available assignment including difficult tasks,such as lifting heavy items, performing a long duration task, orperforming many assignments within a retail store in a single stop.

The reward may be provided prior to the user accepting an assignment.For instance, server 135 may provide to the user an indication of areward corresponding to the external assignment without the selected atleast one available assignment and an indication of a rewardcorresponding to a bundle of the external assignment and the selected atleast one available assignment. In this way, the reward may incentivizethe user to accept the offered bundle of the external assignment and theselected at least one available assignment.

In some embodiments, retail stores may submit assignments to server 135for distribution to users. Additionally or alternatively, server 135 maygenerate assignments based, for instance, on images captured from withina retail store. For example, cameras in a retail store may indicate thata particular item is not displayed according to a planogram. A marketingmanager may desire to have a human take a picture of the items in theretail store to record what a human views, as opposed to installedcameras, thereby providing additional data to remediate the improperdisplay.

Accordingly, server 135 may receive one or more images captured from theselected retail store, such as from users, installed cameras, and/ormobile cameras. Server 135 may also analyze the one or more images toselect at least one available assignment. For example, at least oneconvolution of at least part of the one or more images may becalculated, and the at least one available assignment may be selectedbased on the calculated at least one convolution. For instance, inresponse to a first value of a convolution of the at least part of theone or more images, server 135 may select a first at least one availableassignment. Further, in response to a second value of the convolution ofthe at least part of the one or more images, server 135 may select asecond at least one available assignment, and the second at least oneavailable assignment may differ from the first at least one availableassignment.

To further illustrate, server 135 may store an image of a partiallyempty shelf of a grocery store. A camera in the grocery store maycapture an image of the shelf, and server 135 may convolve the capturedimage with the image of the partially empty shelf. Based on a value ofthe convolution, server 135 may generate an assignment to take a pictureof the shelf, and offer the assignment to a user through process 4100.

In some embodiments, server 135 may also determine compatibility basedon historical assignments provided to a user. The historical assignmentsand an assignment in the selected retail store may have a matchingcategory, such as image capture, or a common location, such as aparticular store. Further, server 135 may access a success indicationcorresponding to a historical assignments of a user. Server 135 may thenuse the success indication to determine whether a new assignment shouldbe offered to the user. Accordingly, server 135 may avoid offeringassignments to a user that are similar to assignments previously failedby the user.

In some cases, a user may decline an assignment due to time constraints,disinterest, or an inadequate incentive. Accordingly, server 135 mayreceive an indication that the user has declined the selected at leastone available assignment. The indication may also provide a reason thatthe user declined the selected at least one available assignment, suchas a dangerous location of the retail store, or the retail storerequiring the user to walk farther than the user desires. Server 135 mayalso select an alternate retail store of among the plurality of retailstores, and, based on the selected alternate retail store, select atleast one alternate available assignment in the selected alternateretail store. Further, server 135 may offer the selected at least onealternate available assignment in the selected alternate retail store tothe user. In some embodiments, an incentive or payment for the alternateretail store may be decreased to disincentive a user from decliningoffered assignments.

Server 135 may also determine a category of a declined assignment andoffer alternative assignments of a different category. For example, auser may decline picking up goods from a drive through, and server 135may proceed to select a new assignment that does not require accessing adrive through. In this manner, server 135 may avoid annoyance and missedproductivity of a particular user.

The foregoing description has been presented for purposes ofillustration. It is not exhaustive and is not limited to the preciseforms or embodiments disclosed. Modifications and adaptations will beapparent to those skilled in the art from consideration of thespecification and practice of the disclosed embodiments. Additionally,although aspects of the disclosed embodiments are described as beingstored in memory, one skilled in the art will appreciate that theseaspects can also be stored on other types of computer readable media,such as secondary storage devices, for example, hard disks or CD ROM, orother forms of RAM or ROM, USB media, DVD, Blu-ray, 4K Ultra HD Blu-ray,or other optical drive media.

Computer programs based on the written description and disclosed methodsare within the skill of an experienced developer. The various programsor program modules can be created using any of the techniques known toone skilled in the art or can be designed in connection with existingsoftware. For example, program sections or program modules can bedesigned in or by means of .Net Framework, .Net Compact Framework (andrelated languages, such as Visual Basic, C, etc.), Java, C++,Objective-C, HTML, HTML/AJAX combinations, XML, or HTML with includedJava applets.

Moreover, while illustrative embodiments have been described herein, thescope of any and all embodiments having equivalent elements,modifications, omissions, combinations (e.g., of aspects across variousembodiments), adaptations and/or alterations as would be appreciated bythose skilled in the art based on the present disclosure. Thelimitations in the claims are to be interpreted broadly based on thelanguage employed in the claims and not limited to examples described inthe present specification or during the prosecution of the application.The examples are to be construed as non-exclusive. Furthermore, thesteps of the disclosed methods may be modified in any manner, includingby reordering steps and/or inserting or deleting steps. It is intended,therefore, that the specification and examples be considered asillustrative only, with a true scope and spirit being indicated by thefollowing claims and their full scope of equivalents.

What is claimed is:
 1. A non-transitory computer-readable mediumincluding instructions that when executed by a processor cause theprocessor to perform a method for identifying products from on-shelfsensors and image data, the method comprising: receiving data capturedusing a plurality of sensors positioned between at least part of aretail shelf and one or more products placed on the at least part of theretail shelf; receiving an image of the at least part of the retailshelf and at least one of the one or more products; and analyzing thecaptured data and the image to determine a product type of the one ormore products.
 2. The non-transitory computer-readable medium of claim1, wherein the one or more products include at least one product notdepicted in the image.
 3. The non-transitory computer-readable medium ofclaim 1, wherein the captured data is indicative of a weight of aproduct among the one or more products.
 4. The non-transitorycomputer-readable medium of claim 1, wherein the captured data isindicative of a pressure caused by a product among the one or moreproducts.
 5. The non-transitory computer-readable medium of claim 1,wherein the captured data is indicative of a footprint of a productamong the one or more products.
 6. The non-transitory computer-readablemedium of claim 1, wherein the captured data is indicative of ambientlight captured by at least one of the plurality of sensors.
 7. Thenon-transitory computer-readable medium of claim 1, wherein the imageincludes depth information of the at least one of the one or moreproducts.
 8. The non-transitory computer-readable medium of claim 1,wherein the method further comprises analyzing the captured data and theimage to determine a quantity of the one or more products.
 9. Thenon-transitory computer-readable medium of claim 1, wherein the methodfurther comprises analyzing the captured data and the image to determinefacings of the one or more products.
 10. The non-transitorycomputer-readable medium of claim 1, wherein the method furthercomprises analyzing the captured data and the image to determine qualityof the one or more products.
 11. The non-transitory computer-readablemedium of claim 1, wherein the method further comprises: analyzing thecaptured data and the image to determine an action associated with theretail shelf; and providing information configured to cause theperformance of the action.
 12. The non-transitory computer-readablemedium of claim 1, wherein the method further comprises: analyzing thecaptured data to determine a need to capture visual data; and inresponse to the determined need to capture visual data, triggering thecapture of the image.
 13. The non-transitory computer-readable medium ofclaim 1, wherein the method further comprises: analyzing the captureddata to determine a plurality of alternative candidate product types;and analyzing the image to select the product type of the one or moreproducts from the plurality of alternative candidate product types. 14.The non-transitory computer-readable medium of claim 1, wherein themethod further comprises: analyzing the image to determine a pluralityof alternative candidate product types; and analyzing the captured datato select the product type of the one or more products from theplurality of alternative candidate product types.
 15. The non-transitorycomputer-readable medium of claim 1, wherein analyzing the captured dataand the image to determine the product type of the one or more productscomprises: analyzing the image to determine a first product typeassociated with the one or more products; analyzing the captured data todetermine a second product type associated with the one or moreproducts; and determining the product type of the one or more productsbased on a comparison of the first and second product types.
 16. Thenon-transitory computer-readable medium of claim 1, wherein analyzingthe captured data and the image to determine the product type of the oneor more products comprises: extracting one or more features from thecaptured data and one or more features from the image; analyzing the oneor more extracted features from the captured data and the one or morefeatures from the image using a product recognition model; anddetermining the product type of the one or more products based on anoutput of the product recognition model.
 17. The non-transitorycomputer-readable medium of claim 1, wherein analyzing the captured dataand the image to determine the product type of the one or more productscomprises: calculating at least one convolution of at least part of theimage; and analyzing the calculated at least one convolution of at leastpart of the image and the captured data to determine the product type ofthe one or more products.
 18. The non-transitory computer-readablemedium of claim 1, wherein the captured data includes at least an arrayof values, and wherein analyzing the captured data and the image todetermine the product type of the one or more products comprises:calculating at least one convolution of at least part of the image;calculating at least one convolution of at least part of the array ofvalues; and analyzing the calculated at least one convolution of atleast part of the image and the at least one convolution of the array ofvalues to determine the product type of the one or more products.
 19. Amethod for identifying products from on-shelf sensors and image data,the method comprising: receiving data captured using a plurality ofsensors positioned between at least part of a retail shelf and one ormore products placed on the at least part of the retail shelf; receivingan image of the at least part of the retail shelf and at least one ofthe one or more products; and analyzing the captured data and the imageto determine a product type of the one or more products.
 20. A systemfor identifying products from on-shelf sensors and image data, thesystem comprising: at least one processor programmed to: receive datacaptured using a plurality of sensors positioned between at least partof a retail shelf and one or more products placed on the at least partof the retail shelf; receive an image of the at least part of the retailshelf and at least one of the one or more products; and analyze thecaptured data and the image to determine a product type of the one ormore products.